I ran a query on the Yahoo Screener at: https://finance.yahoo.com/screener/equity/new DevTools shows that the data came back as JSON via: https://query2.finance.yahoo.com/v1/finance/screener?crumb=u0eNvTHfT6U&lang=en-US®ion=US&formatted=true&corsDomain=finance.yahoo.com So I tried to manually request the data with: import json import requests url = “https://query2.finance.yahoo.com/v1/finance/screener” payload = json.loads(‘{“size”:25,”offset”:0,”sortField”:”intradaymarketcap”,”sortType”:”DESC”,”quoteType”:”EQUITY”,”topOperator”:”AND”,”query”:{“operator”:”AND”,”operands”:[{“operator”:”or”,”operands”:[{“operator”:”EQ”,”operands”:[“region”,”us”]}]},{“operator”:”or”,”operands”:[{“operator”:”LT”,”operands”:[“intradaymarketcap”,2000000000]},{“operator”:”BTWN”,”operands”:[“intradaymarketcap”,2000000000,10000000000]}]}]},”userId”:””,”userIdType”:”guid”}’) header = { “authority”: “query2.finance.yahoo.com”, “method”:”POST”, “path”:”/v1/finance/screener?crumb=umZV3T8[ETC…]&lang=en-US®ion=US&formatted=true&corsDomain=finance.yahoo.com”, “scheme”:”https”, “Accept”:”*/*”, “Accept-Encoding”:”gzip, deflate, br”, “Accept-Language”:”en-US,en;q=0.9″, “Access-Control-Request-Headers”:”content-type”, “Access-Control-Request-Method”:”POST”, … Read more