Yahoo Finance Futures Contracts Historical Data

Futures data downloaded from yahoo finance are not adjusted as continuous contracts. When you download futures data from Yahoo Finance or many other financial data sources, the data is typically provided on a contract-by-contract basis, and it is not automatically adjusted as continuous contracts. Continuous contracts are constructed by rolling over from one contract to the next when the current contract expires. To create a continuous contract from individual contract data, you'll need to implement a custom rolling algorithm that accounts for the roll date and the price adjustment. The main steps involved in creating a continuous contract are as follows: 

 - Data Retrieval: Obtain historical futures data for each individual contract from a data source. 

 - Roll Date Detection: Determine the roll dates, which indicate when you should switch from one contract to the next. Roll dates can be based on various criteria, such as volume, open interest, or a specific calendar date. 

 - Price Adjustment: Apply price adjustments to account for the difference between the expiring contract and the new one. 

 Common price adjustments include:

- Percentage Price Adjustment: Adjust prices based on the percentage change in the new contract's settlement price compared to the old contract's settlement price. 

- Price Ratio Adjustment: Adjust prices based on the ratio of the new contract's price to the old contract's price. 

 - Dollar Value Adjustment: Adjust prices based on the change in dollar value between the two contracts. 

 - Combine Data: Combine the adjusted data from each individual contract to create a continuous price series. 

 Constructing continuous contracts is a non-trivial task and can be complex due to variations in contract specifications, roll rules, and price adjustments for different futures markets. 

There are software libraries and tools available that can assist in this process, such as QuantLib and the Python library roll-contract-continuous. 

 Additionally, you may consider using dedicated data providers or services that offer continuous contract data to save you the effort of manually constructing them. These services often provide accurate and reliable continuous contract data for various futures markets. 

  @QuantScraper #Futures #StockMarket

Python: Regularities at beginning/end of month

An example in Python to create 2 columns: - ascending from 1st day of month - and descending from last day of month using intraday data 10 minutes timeframe. Useful to explore regularities around end/beginning of month. I don't know if there is a more efficient way of doing this. If anyone has a proposal please share. After many and many tentatives ChatGPT did it almost right....😁 #Pyhton #pythonprogramming #100DaysOfCode

First Trading Day Of September - S&P 500

Since 2000 buying the first trading day of September for the Emini S&P has not been bullish over the next 1-5 days. #ES #ES_F #SP500 $ES $SPY $SPX #NQ #QQQ #NQ_F #ZB_F #GC_F #CL_F $EURUSD #VIX

Code To Generate Intraday Test Data

Download Data in Parquet Format

Here you can see how to download a file from Yahoo Finance and save it both in csv and Parquet format. Note that Parquet efficiently compress data to about 60% of its original size. #Python #pythonprogramming

This is also a very good article about How To Efficiently Write Data To Parquet Format.

4 Ways to Write Data To Parquet With Python: A Comparison

@QuantScraper

Lies and Statistics

 “There are three types of lies: lies, damn lies, and statistics…” –Benjamin Disraeli (1804–1881), Prime Minister of Great Britain (1874–1880) #quoteoftheday #quotes #InvestingQuotes @QuantScraper

Python: Parquet - optimized for big data processing

 In Python, Parquet is a columnar storage file format that is designed for efficient data storage and processing. It is optimized for use with big data processing frameworks, such as Apache Hadoop and Apache Spark, but can also be used in standalone Python applications.

The Parquet format offers several advantages over traditional row-based file formats, such as CSV or JSON, especially when working with large datasets:

1. Columnar storage: Parquet stores data column-wise rather than row-wise. This columnar organization allows for more efficient compression and encoding, as similar data values are stored together, reducing storage space and improving query performance.

2. Compression: Parquet supports various compression algorithms, such as Snappy, Gzip, and LZO. Compression helps to reduce the size of the data files, resulting in faster I/O operations and lower storage requirements.

3. Predicate pushdown: Parquet supports predicate pushdown, which means that when executing queries, it can skip reading entire columns or row groups based on the query predicates. This capability improves query performance by minimizing disk I/O.

4. Schema evolution: Parquet files can handle schema evolution, allowing for flexibility in adding, modifying, or deleting columns from the dataset without the need to rewrite the entire dataset.

To work with Parquet files in Python, you can use libraries like pyarrow or pandas that provide convenient APIs for reading and writing Parquet data. These libraries offer methods for converting data between Parquet files and other data structures like DataFrames, enabling seamless integration with existing Python data processing workflows.

@QuantScraper

#Python #pythonprogramming