Yahoo Finance Futures Contracts Historical Data

Futures data downloaded from yahoo finance are not adjusted as continuous contracts. When you download futures data from Yahoo Finance or many other financial data sources, the data is typically provided on a contract-by-contract basis, and it is not automatically adjusted as continuous contracts. Continuous contracts are constructed by rolling over from one contract to the next when the current contract expires. To create a continuous contract from individual contract data, you'll need to implement a custom rolling algorithm that accounts for the roll date and the price adjustment. The main steps involved in creating a continuous contract are as follows: 

 - Data Retrieval: Obtain historical futures data for each individual contract from a data source. 

 - Roll Date Detection: Determine the roll dates, which indicate when you should switch from one contract to the next. Roll dates can be based on various criteria, such as volume, open interest, or a specific calendar date. 

 - Price Adjustment: Apply price adjustments to account for the difference between the expiring contract and the new one. 

 Common price adjustments include:

- Percentage Price Adjustment: Adjust prices based on the percentage change in the new contract's settlement price compared to the old contract's settlement price. 

- Price Ratio Adjustment: Adjust prices based on the ratio of the new contract's price to the old contract's price. 

 - Dollar Value Adjustment: Adjust prices based on the change in dollar value between the two contracts. 

 - Combine Data: Combine the adjusted data from each individual contract to create a continuous price series. 

 Constructing continuous contracts is a non-trivial task and can be complex due to variations in contract specifications, roll rules, and price adjustments for different futures markets. 

There are software libraries and tools available that can assist in this process, such as QuantLib and the Python library roll-contract-continuous. 

 Additionally, you may consider using dedicated data providers or services that offer continuous contract data to save you the effort of manually constructing them. These services often provide accurate and reliable continuous contract data for various futures markets. 

  @QuantScraper #Futures #StockMarket

Python: Regularities at beginning/end of month

An example in Python to create 2 columns: - ascending from 1st day of month - and descending from last day of month using intraday data 10 minutes timeframe. Useful to explore regularities around end/beginning of month. I don't know if there is a more efficient way of doing this. If anyone has a proposal please share. After many and many tentatives ChatGPT did it almost right....😁 #Pyhton #pythonprogramming #100DaysOfCode

First Trading Day Of September - S&P 500

Since 2000 buying the first trading day of September for the Emini S&P has not been bullish over the next 1-5 days. #ES #ES_F #SP500 $ES $SPY $SPX #NQ #QQQ #NQ_F #ZB_F #GC_F #CL_F $EURUSD #VIX

Code To Generate Intraday Test Data

Download Data in Parquet Format

Here you can see how to download a file from Yahoo Finance and save it both in csv and Parquet format. Note that Parquet efficiently compress data to about 60% of its original size. #Python #pythonprogramming

This is also a very good article about How To Efficiently Write Data To Parquet Format.

4 Ways to Write Data To Parquet With Python: A Comparison

@QuantScraper

Lies and Statistics

 “There are three types of lies: lies, damn lies, and statistics…” –Benjamin Disraeli (1804–1881), Prime Minister of Great Britain (1874–1880) #quoteoftheday #quotes #InvestingQuotes @QuantScraper

Python: Parquet - optimized for big data processing

 In Python, Parquet is a columnar storage file format that is designed for efficient data storage and processing. It is optimized for use with big data processing frameworks, such as Apache Hadoop and Apache Spark, but can also be used in standalone Python applications.

The Parquet format offers several advantages over traditional row-based file formats, such as CSV or JSON, especially when working with large datasets:

1. Columnar storage: Parquet stores data column-wise rather than row-wise. This columnar organization allows for more efficient compression and encoding, as similar data values are stored together, reducing storage space and improving query performance.

2. Compression: Parquet supports various compression algorithms, such as Snappy, Gzip, and LZO. Compression helps to reduce the size of the data files, resulting in faster I/O operations and lower storage requirements.

3. Predicate pushdown: Parquet supports predicate pushdown, which means that when executing queries, it can skip reading entire columns or row groups based on the query predicates. This capability improves query performance by minimizing disk I/O.

4. Schema evolution: Parquet files can handle schema evolution, allowing for flexibility in adding, modifying, or deleting columns from the dataset without the need to rewrite the entire dataset.

To work with Parquet files in Python, you can use libraries like pyarrow or pandas that provide convenient APIs for reading and writing Parquet data. These libraries offer methods for converting data between Parquet files and other data structures like DataFrames, enabling seamless integration with existing Python data processing workflows.

@QuantScraper

#Python #pythonprogramming

Python: CPU usage - psutil Library

A working example that compares two dataframes and measures the CPU usage during the comparison:

In this example, we have two example dataframes df and dg. The compare_dataframes function compares the dataframes by using the equals method. You can modify the comparison logic based on your specific requirements.

Before performing the comparison, the initial CPU usage is obtained using psutil.cpu_percent(). After the comparison, the final CPU usage is obtained, and the difference in CPU usage is calculated.

Finally, the result, indicating whether the dataframes are equal, and the CPU usage difference, is printed.

Please note that the CPU usage can vary depending on the specific system specifications and the complexity of the dataframe operations being performed. This example provides a basic approach to measure CPU usage during dataframe comparison, but you may need to adjust it according to your specific use case and requirements.

Following is the result in this case. df was about 3.2Mb and dg which was using specific datatypes was about 1.5Mb.

Dataframes are equal: False
CPU Usage: -1.5%

psutil (python system and process utilities) is a cross-platform library for retrieving information on running processes and system utilization (CPU, memory, disks, network, sensors) in Python. It is useful mainly for system monitoring, profiling, limiting process resources and the management of running processes.

https://psutil.readthedocs.io/en/latest/

S&P: 20-Day High Not So Bulish

 Historical data reveals that a 20-Day High hasn't translated into significant bullishness for the Emini S&P within 1-5 days over the past 3 years. The Figure provides a visual representation. #ES #ES_F #SP500 $ES $SPY $SPX #NQ #QQQ #NQ_F #ZB_F #GC_F #CL_F #eurusd $EURUSD

@QuantScraper

Short term Trading

In short-term trading take what the market gives. Sometimes it's small profits or losses. Occasionally, few windfall trades make a difference. Strategy automatioensures consistent execution and helps capture those opportunities. Stay disciplined, manage risk. #TradingStrategies

When engaging in short-term trading or applying mean reversion setups, it is common to take what the market gives, which can result in small profits or losses. However, occasionally, you may experience windfall profits that can make a significant difference to your overall trading performance.

Consistency is indeed key when trading these strategies, and many traders opt to automate their strategies to ensure consistent execution. Automating your trading system allows you to remove emotional biases and ensures that your strategy is executed according to pre-defined rules. It also enables you to take advantage of opportunities in the market even when you are not actively monitoring it.

By consistently trading your automated strategy, you increase your chances of capturing those few very good trades that can significantly impact your profitability. These trades can help offset smaller losses and generate overall positive returns. However, it's important to note that no trading strategy can guarantee profits, and risk management and thorough backtesting are essential to evaluate the viability of any trading approach.

@QuantScraper

Backtesting Strategies

The Emini S&P Future has printed 5 consecutive times a Down day on Tuesday.  Does it have statistical relevance?

Since 2000 following are results selling the next day:

Not very good......

Regardless the result what are considerations to make sure you are following a reasonable process to assess the relevance of your findings? Let's see how chatgpt answers to the question.

"Determining the statistical relevance of a pattern, such as five consecutive down Tuesdays in the stock market, requires a rigorous statistical analysis. It involves examining historical data, conducting hypothesis testing, and considering various factors such as sample size, significance levels, and the presence of confounding variables".

"It's important to note that isolated patterns or observations, without a solid statistical analysis, may not have significant predictive power. Stock market movements are influenced by a wide range of complex factors, including economic indicators, market sentiment, geopolitical events, and investor behavior. Therefore, relying solely on a short-term pattern like consecutive down Tuesdays may not provide reliable predictions or trading strategies".

"To determine the statistical relevance of a pattern like five consecutive down Tuesdays in the stock market, we need to conduct a hypothesis test. Here's a step-by-step approach:

- Formulate the null hypothesis (H0): The occurrence of 5 consecutive down

   Tuesdays in the stock market is due to random chance and has no predictive

    significance.

 - Collect historical data: Gather a dataset containing historical stock market returns

    for Tuesdays over a significant period. Ensure the data is reliable, consistent,

    and covers various market conditions.

- Define the statistical test: Select an appropriate statistical test based on the

   characteristics of the data. One potential test could be a binomial test or a runs

   test, depending on the specific nature of the data.

- Set the significance level (alpha): Determine the acceptable level of risk for

    rejecting the null hypothesis. Commonly used values are 0.05 or 0.01,

    representing a 5% or 1% chance of making a Type I error, respectively.

- Calculate the p-value: Perform the statistical test using the collected data and

   calculate the p-value. The p-value represents the probability of observing the

   pattern (or a more extreme one) assuming the null hypothesis is true.

 - Evaluate the results: Compare the calculated p-value with the significance level. If

   the p-value is smaller than the significance level, we reject the null hypothesis,

   indicating that the pattern is statistically significant. If the p-value is larger than the

   significance level, we fail to reject the null hypothesis, suggesting the pattern may

   be due to chance.

- Consider additional factors: It's crucial to consider the broader context and avoid

   overfitting. Assess whether other variables, such as economic indicators or

   seasonal effects, may be influencing the observed pattern".

Remember, statistical significance does not guarantee practical significance or future predictive power. Even if the pattern is statistically significant, it's essential to conduct further analysis, validate results using out-of-sample data, and consider the limitations and assumptions of the statistical test. These considerations make sense and actually the result provided by my test have no significance in my view mainly because they are based on a very limited number of observations, p-value and T-test are not good and I have not considered using out of sample data.

Software Tool: Python

Backtesting: building your tool offers flexibility and it is cost-effective. You're in control, learning valuable skills. Instead commercial platforms bring efficiency, support, and data integration. It's a trade-off based on goals, expertise, and resources. #Backtesting #Python @QuantScraper

Tradestation: Workspace

I am thrilled with my TradeStation desktop setup! 💻📈 Advanced charting tools, lightning-fast order execution, and a personalized workspace for conquering the futures market. Ready to trade with precision and confidence! 🚀 #TradeStation #TradingTools #FuturesTrader @QuantScraper

I have been using Tradestation for many years now. It has a user friendly environment. The programming language (EasyLanguage) is quite simple to learn. Some advantages of TradeStation desktop setup:

1. TradeStation's desktop platform offers robust charting capabilities with numerous technical indicators, drawing tools, and analytical features. Traders can perform in-depth market analysis.

2. The platform allows you to customize your workspace according to your preferences, including layout, colors, and tool placement, enabling you to create a personalized environment.

3. TradeStation supports automated trading through its proprietary EasyLanguage programming language. You can develop and implement complex trading strategies, backtest them using historical data, and execute trades automatically.

4. It provides fast and reliable order execution, ensuring that trades are executed swiftly to suit your specific trading strategies.

5. TradeStation offers access to extensive market data, including real-time quotes, historical data, and market depth information. This wealth of data empowers you to make trading decisions based on up-to-date information.

At QuantScraper we have developed our own tool in Python to backtest strategies at daily and intraday level. We are continuing to work on our sw because it is perfectly customized for our own needs. It can work with any type of data (including alternative data)nand it fits with our trading style. However, I like Tradestation because it is quite reliable, it has decent fees, you have great access to market data (for us mainly futures), it has a nice programming language, you can manage portfolios or scan the market according to specific filters or indicators.