burakyilmaz321/tefas-crawler

A Python library for crawling public data from Tefas.

Tefas Crawler

Crawl public invenstment fund information from Turkey Electronic Fund Trading Platform (TEFAS) with ease.

Installation

pip install tefas-crawler

Usage

Import the Crawler object and create an instance of it.

from tefas import Crawler

tefas = Crawler()

API

fetch(start, end, name, columns, kind)

Argument	Type	Description	Required
start	string or datetime.datetime	The date that fund information is crawled for.	Yes
end	string or datetime.datetime	End of the period that fund information is crawled for.	No
name	string	Name of the fund. If not given, all funds will be returned.	No
columns[]	list of string	List of columns to be returned.	No
kind	string	Type of the fund. One of YAT, EMK, or BYF. Defaults to YAT.	No

Examples

Get all funds for a given day.

data = tefas.fetch(start="2020-11-20")

Get a specific fund for a time period, and select columns.

data = tefas.fetch(start="2020-11-15", end="2020-11-20", name="YAC", columns=["code", "date", "price"])

Data Schema

As of today, we support the following data schema from Tefas:

Column	Description	Type
date	Sate	date
price	Price of the fund for a given date	string
code	Short code of the fund	string
title	Full name of the fund	string
market_cap	Total value of the fund	float
number_of_shares	Number of outstanding shares	float
number_of_investors	Number of participants	float
bank_bills	Share of bank bills	float
exchange_traded_fund	Share of exchange traded fund	float
other	Share of other	float
fx_payable_bills	Share of fx payable bills	float
government_bond	Share of government bond	float
foreign_currency_bills	Share of foreign currency bills	float
eurobonds	Share of eurobonds	float
commercial_paper	Share of commercial paper	float
fund_participation_certificate	Share of fund participation certificate	float
real_estate_certificate	Share of real estate certificate	float
venture_capital_investment_fund_participation	Share of venture capital investment fund	float
real_estate_investment_fund_participation	Share of real estate investment fund	float
treasury_bill	Share of treasury bill	float
stock	Share of stock	float
government_bonds_and_bills_fx	Share of government bonds and bills (fx)	float
participation_account	Share of participation account	float
participation_account_au	Share of gold participation account	float
participation_account_d	Share of foreign currency participation account	float
participation_account_tl	Share of Turkish Lira participation account	float
government_lease_certificates	Share of government lease certificates	float
government_lease_certificates_d	Share of foreign currency government lease certificates	float
government_lease_certificates_tl	Share of Turkish Lira government lease certificates	float
government_lease_certificates_foreign	Share of government foreign lease certificates	float
precious_metals	Share of precious metals	float
precious_metals_byf	Share of precious metals stock market investment fund	float
precious_metals_kba	Share of precious metals government dept instrument	float
precious_metals_kks	Share of precious metals public lease certificates	float
public_domestic_debt_instruments	Share of foreign exchange public domestic debt instruments	float
private_sector_lease_certificates	Share of private sector lease certificates	float
private_sector_bond	Share of private sector bond	float
repo	Share of repo	float
derivatives	Share of derivatives	float
tmm	Share of tmm	float
reverse_repo	Share of reverse-repo	float
asset_backed_securities	Share of asset-backed securities	float
term_deposit	Share of term deposit	float
term_deposit_au	Share of gold term deposit	float
term_deposit_d	Share of foreign currency term deposit	float
term_deposit_tl	Share of Turkish Lira term deposit	float
futures_cash_collateral	Share of futures cash collateral	float
foreign_debt_instruments	Share of foreign debt instruments	float
foreign_domestic_debt_instruments	Share of foreign domestic debt instruments	float
foreign_private_sector_debt_instruments	Share of foreign private sector debt instruments	float
foreign_exchange_traded_funds	Share of foreign exchange traded funds	float
foreign_equity	Share of foreign equity	float
foreign_securities	Share of foreign securities	float
foreign_investment_fund_participation_shares	Share of foreign investment fund participation	float
private_sector_international_lease_certificate	Share of private sector international lease certificate	float
private_sector_foreign_debt_instruments	Share of private sector foreign dept instruments	float

To-do

• Increase test coverage
• Request error handling
• Cache query results

License

MIT

tefas-crawler's Issues

No results if date range includes the dates where the fund is not issued yet

Hi,

Thanks for this crawler, it's really helpful.
If you try a date range which includes dates where a fund has not been issued yet, it won't return any results. I think this is about TEFAS limitations as well. You need to know the exact TEFAS issue date for each fund you're pulling the data and adjust it.

Crawler dönen fon sayısı ile fundturkey.com.tr sitesinden export yapılan excel fon sayısı farklı

Merhaba, uygulama için çok teşekkür ederim.

Dönen datanın fon sayısı ile siteden export yapılan fon sayısı arasında farklılık var.

Veride 1042 adet fon varken, excel dosyasında 1130 adet fon var. Aradaki fark 88. Bunların arasından fiyatı olmayanları elediğimizde geriye 18 adet fon kalıyor.

AES AK ASSET MANAGEMENT OIL FOREIGN ETF FUND BASKET FUND
AFA AK ASSET MANAGEMENT AMERICA FOREIGN EQUITY Fund
AFS AK PORTFÖY HEALTH SECTOR FOREIGN SHARE FUND
AFT AK ASSET MANAGEMENT NEW TECHNOLOGIES FOREIGN EQUITY Fund
AFV AK ASSET MANAGEMENT EUROPE FOREIGN EQUITY Fund
AOY AK PORTFÖY ALTERNATIVE ENERGY FOREIGN SHARE FUND
APJ AK PORTFÖY DÖRDÜNCÜ FON SEPETİ FONU
ARL AK ASSET MANAGEMENT FIRST FUND OF FUNDS
ARM AK ASSET MANAGEMENT SECOND FUND OF FUNDS
DTZ AK PORTFÖY DÖNÜŞTÜRÜCÜ TEKNOLOJİLER DEĞİŞKEN FON
GUM AK PORTFÖY GÜMÜŞ FON SEPETI FONU
MTV AK PORTFÖY METAVERSE VE DİJİTAL YAŞAM TEKNOLOJİLERİ DEĞİŞKEN FON
ODV AK ASSET MANAGEMENT THIRD FUND BASKET MUTUAL Fund
PAI AK PORTFÖY ING BANK ÖZEL BANKACILIK VE PLATINUM DEĞİŞKEN ÖZEL FON
TAR AK PORTFÖY TARIM VE GIDA TEKNOLOJİLERİ DEĞİŞKEN FON
TGR AK PORTFÖY TURİZM VE SEYAHAT SEKTÖRÜ DEĞİŞKEN FON
VCY AK PORTFÖY ELEKTRİKLİ VE OTONOM ARAÇ TEKNOLOJİLERİ DEĞİŞKEN FON
ZFB AK PORTFÖY FİNTEK VE BLOKZİNCİRİ TEKNOLOJİLERİ DEĞİŞKEN FON

Hepsi Ak Portföy kağıtlarından.

Fon adet farklılığı neden kaynaklanmış olabilir?

Fon ünvan tipi ve Türkçe Tanım

Fon Ünvan tipi (Şemsiye, ParaPiyasası vs..) bilgisi de çekilebilirse çok güzel olur.
Ayrıca fonların isimleri bazılarında İngilizce bazılarında Türkçe geliyor, dil seçimi de koyabilir misiniz?

ssl error

selamlar,
farklı zamanlarda denememe rağmen aşağıdaki hatayı alıyorum, yardımcı olabilir misin?

SSLError: HTTPSConnectionPool(host='www.tefas.gov.tr', port=443): Max retries exceeded with url: /TarihselVeriler.aspx (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1002)')))

Optional Fund Profit Comparison Endpoint Support

Hi, maybe support for fund returns comparison page can be added.

Post data sample:

        url=f"https://fundturkey.com.tr/api/DB/BindComparisonFundReturns",
        data= {
            "calismatipi": 1,
            "fontip": "YAT",
            "bastarih": "14.01.2024",
            "bittarih": "16.01.2024",
            "islemdurum": 1  
        }

Response sample:

    {
        "FONKODU": "YUN",
        "FONUNVAN": "YAPI KRED\u0130 PORTF\u00d6Y \u0130ST\u0130NYE SERBEST (D\u00d6V\u0130Z) FON",
        "FONTURACIKLAMA": "Hedge Umbrella Fund",
        "GETIRIORANI": 0.2569
    },
    {
        "FONKODU": "YZK",
        "FONUNVAN": "YAPI KRED\u0130 PORTF\u00d6Y KALAMI\u015e SERBEST FON",
        "FONTURACIKLAMA": "Hedge Umbrella Fund",
        "GETIRIORANI": 0.0964
    }

OR

        url=f"https://fundturkey.com.tr/api/DB/BindComparisonFundReturns",
        data= {
            "calismatipi": 2,
            "fontip": "YAT",
            "strperiod": "1,1,0,1,1,0,0",
            "islemdurum": 1
        }

    {
        "FONKODU": "YJH",
        "FONUNVAN": "YAPI KRED\u0130 PORTF\u00d6Y TEM\u0130Z ENERJ\u0130 DE\u011e\u0130\u015eKEN FONU",
        "FONTURACIKLAMA": "Variable Umbrella Fund",
        "GETIRI1A": -2.2096,
        "GETIRI3A": 10.8607,
        "GETIRI6A": null,
        "GETIRI1Y": 19.2132,
        "GETIRIYB": -5.4741,
        "GETIRI3Y": null,
        "GETIRI5Y": null
    },
    {
        "FONKODU": "YPV",
        "FONUNVAN": "YAPI KRED\u0130 PORTF\u00d6Y \u00dc\u00c7\u00dcNC\u00dc FON SEPET\u0130 FONU",
        "FONTURACIKLAMA": "Fund Of Funds Umbrella Fund",
        "GETIRI1A": 0.7878,
        "GETIRI3A": 10.2034,
        "GETIRI6A": null,
        "GETIRI1Y": 53.4969,
        "GETIRIYB": 2.6137,
        "GETIRI3Y": null,
        "GETIRI5Y": null
    }

islemdurum denotes if the fund is traded in Tefas. Also I would really appreciate if there was a way to make the same distinction for the historical data endpoint.

Adding pension funds to Crawler

Changing "fontip" as "EMK" in Crawler will result in crawling Individual Pension System funds as well. Can we make it optional in Crawler?

Any request longer than 3 months returns empty dataframe.

Hello, first of all thank you very much for this crawler.

If you try to fetch data with a time window that is longer than 3 months, the response will return an empty dataframe. This is due to Tefas API not allowing any such queries.

Maybe a check will make the code more robust, I am currently fetching data chunk by chunk.

Anyways thanks again.

Make portfolio breakdown optional

Crawler.fetch gets data from "/api/DB/BindHistoryAllocation" and "/api/DB/BindHistoryInfo" endpoints. Making BindHistoryAllocation optional could make things faster.