diff --git a/surfsense_backend/tests/unit/scrapers/reddit/fixtures/sample_listing.json b/surfsense_backend/tests/unit/scrapers/reddit/fixtures/sample_listing.json
new file mode 100644
index 000000000..d984552a9
--- /dev/null
+++ b/surfsense_backend/tests/unit/scrapers/reddit/fixtures/sample_listing.json
@@ -0,0 +1 @@
+{"kind": "Listing", "data": {"after": "t3_1ugsdwl", "dist": 25, "modhash": "", "geo_filter": null, "children": [{"kind": "t3", "data": {"approved_at_utc": null, "subreddit": "Python", "selftext": "Post all of your code/projects/showcases/AI slop here. \n\nRecycles once a month.", "author_fullname": "t2_6l4z3", "saved": false, "mod_reason_title": null, "gilded": 0, "clicked": false, "title": "Showcase Thread", "link_flair_richtext": [{"e": "text", "t": "Showcase"}], "subreddit_name_prefixed": "r/Python", "hidden": false, "pwls": 6, "link_flair_css_class": "showcase", "downs": 0, "thumbnail_height": null, "top_awarded_type": null, "hide_score": false, "name": "t3_1tws1w7", "quarantine": false, "link_flair_text_color": "light", "upvote_ratio": 0.89, "author_flair_background_color": null, "subreddit_type": "public", "ups": 29, "total_awards_received": 0, "media_embed": {}, "thumbnail_width": null, "author_flair_template_id": null, "is_original_content": false, "user_reports": [], "secure_media": null, "is_reddit_media_domain": false, "is_meta": false, "category": null, "secure_media_embed": {}, "link_flair_text": "Showcase", "can_mod_post": false, "score": 29, "approved_by": null, "is_created_from_ads_ui": false, "author_premium": true, "thumbnail": "self", "edited": false, "author_flair_css_class": null, "author_flair_richtext": [], "gildings": {}, "content_categories": null, "is_self": true, "mod_note": null, "created": 1780589106.0, "link_flair_type": "richtext", "wls": 6, "removed_by_category": null, "banned_by": null, "author_flair_type": "text", "domain": "self.Python", "allow_live_comments": false, "selftext_html": "Post all of your code/projects/showcases/AI slop here.
\n\n
Recycles once a month.
\n
Weekly Thread: Resource Request and Sharing \ud83d\udcda
\n\n
Stumbled upon a useful Python resource? Or are you looking for a guide on a specific topic? Welcome to the Resource Request and Sharing thread!
\n\n
How it Works:
\n\n
\n- Request: Can't find a resource on a particular topic? Ask here!
\n- Share: Found something useful? Share it with the community.
\n- Review: Give or get opinions on Python resources you've used.
\n
\n\n
Guidelines:
\n\n
\n- Please include the type of resource (e.g., book, video, article) and the topic.
\n- Always be respectful when reviewing someone else's shared resource.
\n
\n\n
Example Shares:
\n\n
\n- Book: "Fluent Python" - Great for understanding Pythonic idioms.
\n- Video: Python Data Structures - Excellent overview of Python's built-in data structures.
\n- Article: Understanding Python Decorators - A deep dive into decorators.
\n
\n\n
Example Requests:
\n\n
\n- Looking for: Video tutorials on web scraping with Python.
\n- Need: Book recommendations for Python machine learning.
\n
\n\n
Share the knowledge, enrich the community. Happy learning! \ud83c\udf1f
\n
I have this really messy code scattered in 5 files, it s not that long abt 4000 lines.
\nadding a new feature started feeling like a pain,
\nI hate to deliver this tomorrow, and it s not meant to be scalable.
\nshould i refactor or add the features ?
\n\n
or just add the features and keep the code structure unchanged
\n
I wrote a practical article about FastAPI's app.frontend() feature.
\n\n
The interesting bit is that it serves static frontend build output as low-priority routes, so normal FastAPI API endpoints still win.
\n\n
The article covers:
\n\n
\napp.frontend("/", directory="dist")- SPA fallback with
fallback="index.html" \n- how it differs from
StaticFiles \n- serving under a prefix with
APIRouter \n- a complete mini dashboard example with FastAPI + vanilla JS
\n
\n
Weekly Thread: Meta Discussions and Free Talk Friday \ud83c\udf99\ufe0f
\n\n
Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!
\n\n
How it Works:
\n\n
\n- Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
\n- Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
\n- News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.
\n
\n\n
Guidelines:
\n\n
\n- All topics should be related to Python or the /r/python community.
\n- Be respectful and follow Reddit's Code of Conduct.
\n
\n\n
Example Topics:
\n\n
\n- New Python Release: What do you think about the new features in Python 3.11?
\n- Community Events: Any Python meetups or webinars coming up?
\n- Learning Resources: Found a great Python tutorial? Share it here!
\n- Job Market: How has Python impacted your career?
\n- Hot Takes: Got a controversial Python opinion? Let's hear it!
\n- Community Ideas: Something you'd like to see us do? tell us.
\n
\n\n
Let's keep the conversation going. Happy discussing! \ud83c\udf1f
\n
Running Celery on AWS ECS can be trickier than it seems if you want to avoid lost tasks and ensure all work is completed. Especially if you're frequently deploying to production and using autoscaling.
\n\n
There are two main components for reliable processing:\n- Celery configuration updates\n- Structuring tasks
\n\n
For Celery, you should update the following settings:\n- task_acks_late -> True: To treat tasks as successfully processed only after processing. Otherwise, tasks are not retried.\n- task_reject_on_worker_lost -> True: To ensure tasks are retried if workers die for any reason (e.g., warm shutdown + SIGKILL).\n- worker_prefetch_multiplier -> 1: To avoid unnecessarily delayed tasks.\n- broker_connection_retry_on_startup -> True: To make startups more reliable.\n- broker_transport_options -> {"confirm_publish": True}: To avoid unsubmitted tasks due to message transport issues.\n- Make sure exponential retries are enabled. This way, you ensure that tasks are retried in the event of an interruption.
\n\n
For structuring tasks, use the following two approaches:\n- Batching: Instead of doing all the work at once, you split the work into batches. e.g., Process 1000 users, then submit the next job to process the next 1000 users.\n- Fan out: You can split the work between a "scheduler" task and "execution" tasks. e.g., One task to list all the users and submit email sending tasks, another task to actually send an email for the selected user
\n\n
The same applies to other similar services, such as Heroku and Azure App Containers, which use short grace periods during rolling deployments and downscaling.
\n\n
You can read a more elaborate tutorial here: https://jangiacomelli.com/blog/celery-on-aws-ecs/
\n
Weekly Thread: Professional Use, Jobs, and Education \ud83c\udfe2
\n\n
Welcome to this week's discussion on Python in the professional world! This is your spot to talk about job hunting, career growth, and educational resources in Python. Please note, this thread is not for recruitment.
\n\n
\n\n
How it Works:
\n\n
\n- Career Talk: Discuss using Python in your job, or the job market for Python roles.
\n- Education Q&A: Ask or answer questions about Python courses, certifications, and educational resources.
\n- Workplace Chat: Share your experiences, challenges, or success stories about using Python professionally.
\n
\n\n
\n\n
Guidelines:
\n\n
\n- This thread is not for recruitment. For job postings, please see r/PythonJobs or the recruitment thread in the sidebar.
\n- Keep discussions relevant to Python in the professional and educational context.
\n
\n\n
\n\n
Example Topics:
\n\n
\n- Career Paths: What kinds of roles are out there for Python developers?
\n- Certifications: Are Python certifications worth it?
\n- Course Recommendations: Any good advanced Python courses to recommend?
\n- Workplace Tools: What Python libraries are indispensable in your professional work?
\n- Interview Tips: What types of Python questions are commonly asked in interviews?
\n
\n\n
\n\n
Let's help each other grow in our careers and education. Happy discussing! \ud83c\udf1f
\n
A tip that's saved us a lot of boilerplate across our Python stack (Litestar, and our document-extraction tooling): stop decoding JSON into dict[str, Any] and casting/.get()-ing your way through it. Decode straight into your declared type.
\n\n
msgspec validates and decodes directly into your type at C speed. Quick comparison of the usual options on the same payload:
\n\n
\njson.loads / orjson.loads -> dict[str, Any] (cast and pray; orjson just faster)pydantic TypeAdapter(...).validate_json -> your model, validated + rich, but heaviermsgspec.json.decode(raw, type=T) -> your type, validated, C-fast
\n\n
pydantic does far more and its Rust core is fast; for model-heavy code it's still my default. But on hot paths where you just need decode-into-a-struct, a C decoder going straight to the type is hard to beat.
\n\n
With PEP 695 generics the whole (de)serialization layer collapses to one function:
\n\n
```python\ndef deserialize[T](raw: bytes, t: type[T]) -> T:\n return msgspec.json.decode(raw, type=t, strict=False)
\n\n
deserialize(raw, Grant) # -> Grant\ndeserialize(raw, list[Grant]) # -> list[Grant]\n```
\n\n
We landed on this while building Litestar (msgspec is a big reason it's fast) and reuse it across everything now. How do you handle hot-path decoding \u2014 msgspec, orjson + manual validation, or full pydantic?
\n
I\u2019ve been using the Pythonista IDE by Ole Zorn for over 10 years and I\u2019m just amazed at how consistently good it is. It doesn\u2019t have the latest greatest features but I still use it almost daily. Works with IOS Shortcuts as well. This would be a good one to add to the Wiki.
\n
The clustering problem with correlated signals
\n\n
My system scores ~40 macro signals (Fed funds rate, yield curve, M2, insider buying, short interest, etc.) and generates a composite "confluence score" for a given ticker. The naive approach is to just average the signals. Problem: many signals are correlated \u2014 yield curve and credit spreads move together, insider buying and short interest are often inversely related. Averaging them inflates apparent confidence.
\n\n
Fix I landed on: pairwise Pearson correlation matrix using pandas + numpy on 3 years of weekly signal history. Then scipy.cluster.hierarchy.linkage with single-linkage at a 0.6 threshold groups correlated signals into clusters. Each cluster gets one vote, weighted by the cluster member with the best out-of-sample Sharpe ratio on that ticker's 60-day forward returns.
\n\n
Streamlit caching gotchas
\n\n
@st.cache_data is great but has a subtle memory issue: it keeps ALL cached versions until max_entries is hit. For a function that fetches 40 signals with 5 time-period variations, you can end up caching 200+ DataFrames. Added max_entries=1 to the main signals cache \u2014 memory dropped from ~1.1GB to ~200MB under concurrent load.
\n\n
Also: calling ThreadPoolExecutor inside a cached function is fine for pure data fetching. But if the cached function spawns threads that themselves call other cached functions, you can hit Streamlit's session state lock. Solution: only parallelize at the outermost uncached layer.
\n\n
SEC EDGAR Form 4 XML parsing
\n\n
EDGAR serves Form 4 filings as XML, but namespace handling is inconsistent across filings. Some have explicit xmlns declarations, some don't. I strip namespaces with a regex before parsing:
\n\n
xml_str = re.sub(r'\\s*xmlns[^"]*"[^"]*"', '', raw_xml)\ntree = ET.fromstring(xml_str)\n
\n\n
For insider cluster detection (flagging when 2+ insiders buy within 21 days), I group by issuer CIK, filter for transactionCode == 'P' (open-market purchase), then use a rolling window on sorted transaction dates.
\n\n
SQLAlchemy Core schema
\n\n
Using SQLAlchemy Core (not ORM) for the main tables: users, signal_snapshots, watchlist_items, alerts. One thing I'm glad I did: a single DATABASE_URL env var that switches between Postgres (prod) and SQLite (local dev). Same schema DDL works for both \u2014 keeps the local dev loop fast.
\n\n
Happy to answer questions on any of the above.
\n
I saw one yesterday during an interview and it really confused me at first since the feature has been deprecated for so long.
\n\n
Are there still code bases out there running Python 2? I used Python 3.8 at my last job and that made me feel like a dinosaur.
\n
I've been experimenting with AI agents and autocomplete platforms for a greenfield FastAPI project. In the first few weeks, it felt incredibly fast. But now that we've scaled to multiple routers, complex Pydantic schemas, and SQLAlchemy models, the structural debt is piling up.
\n\n
The AI writes code that functions, but it constantly violates our architecture. It'll put complex business logic inside a route handler instead of the service layer, or it'll mess up async database sessions across modules. I find myself spending more time refactoring the structure of what it built than it would have taken to write the logic myself.
\n\n
Is anyone else hitting this scaling wall where AI utility drops off as codebase complexity grows? How are you keeping your system architecture clean?
\n
I just spent three hours trying to get a basic python cron script to send out a weekly web scraping summary. used to just use smtplib and a random gmail app password but google basically killed that workflow
\n\n
Tried installing the official python sdk for one of the big email providers and it pulled in like 6 different async dependencies just to send a plain text string. It is genuinely insane how bloated the modern python ecosystem has gotten for the most basic tasks
\n\n
I ended up just writing a simple requests.post() webhook over to yaplet to handle the actual subscriber list and formatting because I absolutely refuse to fight with another bloated __init__.py or dns auth protocol this month
\n\n
sometimes it really feels like we spend 10% of our time writing actual python logic and 90% fighting with enterprise api wrappers tbh
\n
This is the first part of a multi-part series exploring why async/await might not be the best concurrency pattern for most use cases, and what alternative models you should consider instead. Using Python for our practical examples, this opening post digs into the roots of async/await, guiding you through building a custom event loop from scratch using generators.
\n\n
https://theblog.info/posts/asyncawait-is-a-plague-part-1-roots
\n\n
Note: This is Part 1 of a multi-part series. Instead of diving straight into why async/await can be problematic, this post explores the original motivations behind the pattern. Understanding how it works under the hood will provide the essential context for the issues we'll discuss in upcoming parts.
\n
Weekly Wednesday Thread: Advanced Questions \ud83d\udc0d
\n\n
Dive deep into Python with our Advanced Questions thread! This space is reserved for questions about more advanced Python topics, frameworks, and best practices.
\n\n
How it Works:
\n\n
\n- Ask Away: Post your advanced Python questions here.
\n- Expert Insights: Get answers from experienced developers.
\n- Resource Pool: Share or discover tutorials, articles, and tips.
\n
\n\n
Guidelines:
\n\n
\n- This thread is for advanced questions only. Beginner questions are welcome in our Daily Beginner Thread every Thursday.
\n- Questions that are not advanced may be removed and redirected to the appropriate thread.
\n
\n\n
Recommended Resources:
\n\n
\n\n
Example Questions:
\n\n
\n- How can you implement a custom memory allocator in Python?
\n- What are the best practices for optimizing Cython code for heavy numerical computations?
\n- How do you set up a multi-threaded architecture using Python's Global Interpreter Lock (GIL)?
\n- Can you explain the intricacies of metaclasses and how they influence object-oriented design in Python?
\n- How would you go about implementing a distributed task queue using Celery and RabbitMQ?
\n- What are some advanced use-cases for Python's decorators?
\n- How can you achieve real-time data streaming in Python with WebSockets?
\n- What are the performance implications of using native Python data structures vs NumPy arrays for large-scale data?
\n- Best practices for securing a Flask (or similar) REST API with OAuth 2.0?
\n- What are the best practices for using Python in a microservices architecture? (..and more generally, should I even use microservices?)
\n
\n\n
Let's deepen our Python knowledge together. Happy coding! \ud83c\udf1f
\n
We compared the 3 pip-installable mesh boolean libraries on the task of pairwise mesh booleans at industry scale.
\n\n
Full write up: https://polydera.com/algorithms/python-mesh-boolean-libraries-2026
\n\n
Libraries tested
\n\n
\n- MeshLib 3.1 \u2014
pip install meshlib. Simulation of Simplicity for degeneracy handling. \n- Manifold 3.5 \u2014
pip install manifold3d. Deterministic floating point with symbolic perturbation. \n- trueform 0.9.8 \u2014
pip install trueform. Topologically-exact arrangements via a bounded integer kernel. Ships native, python and WebAssembly. \n
\n\n
All three are native cores (C++/Rust) with Python bindings.
\n\n
Protocol
\n\n
Each library is timed from input arrays (vertices, triangles) to output arrays of the same shape. Native-object construction \u2014 acceleration structures, trees, topology \u2014 plus the boolean, all in the timer. Only file I/O is outside. Best of 5, fresh objects every run; nothing amortised across calls.
\n\n
Result agreement. On every pair the three produced the same solid \u2014 signed volumes agree within floating-point tolerance. trueform and Manifold returned a closed, manifold mesh on all 1000 pairs; MeshLib on 999. The comparison is wall-clock only.
\n\n
Corpus. Random sets of solid, manifold, non-self-intersecting Thingi10K meshes, 200K to 1.5M polygons per operand. Each operand is normalised to unit extent, randomly rotated, and translated so the bounding boxes overlap; each pair takes the union. Thingi10K IDs and per-operand counts for every case are published: pairwise corpus.
\n\n
Environment. Apple M4 Max (arm64), macOS, CPython 3.13. Installed from PyPI: trueform 0.9.8, meshlib 3.1.0.75, manifold3d 3.5.1 \u2014 default builds, default thread count. On Apple Silicon the wheel's compiled architecture matters; all three ship native arm64 builds.
\n\n
Results
\n\n
Of the libraries you can pip install, trueform was the fastest mesh boolean in Python \u2014 fastest on every one of the 1000 pairwise pairs.
\n\n
Pairwise \u2014 one boolean per pair across the 1000-pair corpus.
\n\n
\n\n| library | \nmedian (ms) | \ngeomean \u00d7 vs trueform | \nvalid / 1000 | \n
\n\n\n| trueform 0.9.8 | \n18.0 | \n1.0\u00d7 | \n1000 | \n
\n\n| MeshLib 3.1 | \n87.6 | \n4.9\u00d7 | \n999 | \n
\n\n| Manifold 3.5 | \n120.3 | \n6.9\u00d7 | \n1000 | \n
\n
\n\n
\n\n
Disclosure: I'm one of the authors of trueform.
\n
Weekly Thread: Project Ideas \ud83d\udca1
\n\n
Welcome to our weekly Project Ideas thread! Whether you're a newbie looking for a first project or an expert seeking a new challenge, this is the place for you.
\n\n
How it Works:
\n\n
\n- Suggest a Project: Comment your project idea\u2014be it beginner-friendly or advanced.
\n- Build & Share: If you complete a project, reply to the original comment, share your experience, and attach your source code.
\n- Explore: Looking for ideas? Check out Al Sweigart's "The Big Book of Small Python Projects" for inspiration.
\n
\n\n
Guidelines:
\n\n
\n- Clearly state the difficulty level.
\n- Provide a brief description and, if possible, outline the tech stack.
\n- Feel free to link to tutorials or resources that might help.
\n
\n\n
Example Submissions:
\n\n
Project Idea: Chatbot
\n\n
Difficulty: Intermediate
\n\n
Tech Stack: Python, NLP, Flask/FastAPI/Litestar
\n\n
Description: Create a chatbot that can answer FAQs for a website.
\n\n
Resources: Building a Chatbot with Python
\n\n
Project Idea: Weather Dashboard
\n\n
Difficulty: Beginner
\n\n
Tech Stack: HTML, CSS, JavaScript, API
\n\n
Description: Build a dashboard that displays real-time weather information using a weather API.
\n\n
Resources: Weather API Tutorial
\n\n
Project Idea: File Organizer
\n\n
Difficulty: Beginner
\n\n
Tech Stack: Python, File I/O
\n\n
Description: Create a script that organizes files in a directory into sub-folders based on file type.
\n\n
Resources: Automate the Boring Stuff: Organizing Files
\n\n
Let's help each other grow. Happy coding! \ud83c\udf1f
\n
Hey r/learnpython \ud83d\udc4b
\n\n
Built PyRun (pyrun.in) \u2014 a browser-based Python learning platform
\n\n
powered by Pyodide. Write and run Python directly in your browser,
\n\n
zero setup needed.
\n\n
What's included:
\n\n
\u2022 Interactive Python editor (Monaco-based)
\n\n
\u2022 Structured lessons from beginner to advanced
\n\n
\u2022 Instant output \u2014 no backend, runs locally in your browser
\n\n
\u2022 Free to use
\n\n
Would love feedback from this community \u2014 what topics or features
\n\n
would make this more useful for you?
\n\n
Link: https://pyrun.in
\n
I'm a coding beginner and after i updated windows the layout started looking different (i can't add pics), i want it to go back to what it used to be as I'm not sure how to use it.
\n
Hey everyone and all space enthusiasts,
\n\n
On April 13, 2029, the asteroid 99942 Apophis will fly past Earth at roughly 38,000 km. Closer than our geostationary satellites! Its discovery history was pretty interesting, because in the beginning there was a 2.7 % chance of an impact (don't worry, it is almost null now). So I decided to use NASA/JPL's SPICE toolkit using Python (spiceypy) to calculate the encounter's properties.
\n\n
I documented the entire project in a Jupyter Notebook and put together a complete video walk-through tutorial. Code: https://github.com/ThomasAlbin/Space-Science-With-Python/blob/main/2026/04_SPICE_Apohis_2029_Flyby.ipynb
\n\n
Video: https://www.youtube.com/watch?v=j4mJTR-BTto
\n\n
What the tutorial does (and why it's 30 minutes long \ud83d\ude05):
\n\n
\n- Computing the time when Apophis enters Earth's gravitational vicinity (the Sphere of Influece)
\n- Computing the closest appraoch (distance and time)
\n- Computing how the orbital elements (the dynamical properties to describe orbits in space) change after the encounter with Earth.
\n
\n\n
By the way: I used also Python + NASA's Cosmographia to create a nice 3D animation of the encounter :). But I will post it soon on my YouTube channel. Since the corresponding code is a complete mess I won't post it here, because I will only post the animation. I have to clean it up...
\n\n
Best,
\n\n
Thomas (your Cassini/Huygens scientist)
\n
Weekly Thread: What's Everyone Working On This Week? \ud83d\udee0\ufe0f
\n\n
Hello r/Python! It's time to share what you've been working on! Whether it's a work-in-progress, a completed masterpiece, or just a rough idea, let us know what you're up to!
\n\n
How it Works:
\n\n
\n- Show & Tell: Share your current projects, completed works, or future ideas.
\n- Discuss: Get feedback, find collaborators, or just chat about your project.
\n- Inspire: Your project might inspire someone else, just as you might get inspired here.
\n
\n\n
Guidelines:
\n\n
\n- Feel free to include as many details as you'd like. Code snippets, screenshots, and links are all welcome.
\n- Whether it's your job, your hobby, or your passion project, all Python-related work is welcome here.
\n
\n\n
Example Shares:
\n\n
\n- Machine Learning Model: Working on a ML model to predict stock prices. Just cracked a 90% accuracy rate!
\n- Web Scraping: Built a script to scrape and analyze news articles. It's helped me understand media bias better.
\n- Automation: Automated my home lighting with Python and Raspberry Pi. My life has never been easier!
\n
\n\n
Let's build and grow together! Share your journey and learn from others. Happy coding! \ud83c\udf1f
\n