Reddit Scraper Returns Different Posts: Why and Fixes

Pull the Reddit posts for a keyword from two sources and see which ones only one source found. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to search Reddit for "rag pipeline evaluation" on two different endpoints, merge the posts on post id, and give me a table showing which posts only one endpoint returned.

One prompt runs the search twice and diffs it. 1 min setup · 3-8 hrs saved.

You ran a Reddit scraper for a keyword, ran it again or tried another one, and got a different list of posts. This guide shows where the difference comes from, measured on 2026-10-16 with one keyword and four calls: the same call twice, the same call with a quoted phrase, and a second provider. It then gives you the merge that turns partial lists into one. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call or per result.

What is the easiest way to scrape Reddit posts for a keyword?

Monid. Install it once, add a key, and your agent calls a Reddit search endpoint with your keyword and gets posts back as JSON, with no browser to run and no scraper to maintain.

• Pay for what you run. The keyword search on TikHub is billed per call, $0.001 whether it returns two posts or twenty (Verified 2026-10-16 on the Reddit tools page). The search on Apify is billed per result, $0.002 a row (Verified 2026-10-16). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free.
• One install, every endpoint. Both searches sit behind the same key and balance, so trying a second source is one more call, not another account.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the tools are there whenever your agent needs them. Your agent reads SKILL.md and learns the rest, so setup is about a minute.

Why does a Reddit scraper return different posts?

For three reasons, and we measured each one on 2026-10-16. The query was rag pipeline evaluation, posts only, sorted by relevance.

Does the same call return the same posts twice?

Yes, within the same minute. We ran the TikHub search twice, two seconds apart, with identical input. Both calls returned the same 7 posts in the same order, starting with t3_1x78yna and ending with t3_1wt1qgk. So a scraper that gives you a different list from one hour to the next is reading a list that changed, not a scraper that guesses. New posts arrive, scores move, and a ranked search over a live site shifts with them. If you need to know whether a change is real, compare lists taken some time apart and keep the older one.

Why does a second provider return a different list?

Because the two sources do not read the same thing. The TikHub endpoint is described in monid inspect as Reddit APP dynamic search, and the Apify actor as a reader of Reddit's official OAuth2 API. We ran the same keyword on both. TikHub returned 7 posts and the Apify actor returned 7 rows (we capped it at maxItems 7). They had zero posts in common. Of the 14 ids, all 14 were different.

Both lists were about RAG. The TikHub list sat mostly in r/Rag (5 of 7), with one post each from r/LangChain and r/AI_Agents. The Apify list had 4 in r/Rag and one each in r/learnmachinelearning, r/AIEducation and r/TechInterviewsIndia. Neither list was wrong. They were two samples of a larger set of matching posts.

Why does quoting the phrase change the list?

Because the query syntax is part of the search. The same endpoint, with the query wrapped in quotes ("rag pipeline evaluation"), returned 7 different posts, and only one id (t3_1wilqoy) matched the unquoted run. The exact phrase appeared in the title of 3 of the 7 posts. The oldest post was from 2024-04-10, where the unquoted run's oldest was from 2026-04-15. A quoted phrase narrows the match and reorders the result, so the quote marks are an input like any other.

So the answer to the complaint is mostly an input question: which source, which query syntax, which sort, and when. The next section is how to fix each of them.

📖 See also Is There an Alternative to Apify for Scraping Reddit?

How do you get the same Reddit posts every time?

Fix the inputs, store every pull with its date, and merge sources on the post id. Setup is below, then the calls.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Search with the per-call endpoint

monid run -p tikhub -e /api/v1/reddit/app/fetch_dynamic_search \
  --query '{"query": "rag pipeline evaluation", "search_type": "post", "sort": "RELEVANCE"}'

Keep sort in uppercase (RELEVANCE, HOT, TOP, NEW or COMMENTS). Add "need_format": true to the query and the response shrinks to children and pageInfo, which is the form we ran. Add "time_range": "week" (all, year, month, week, day or hour) to cap the age of the posts, which is the single best way to make two pulls comparable.

Search with the per-result endpoint

monid run -p apify -e /practicaltools/apify-reddit-api -w \
  -i '{"searches":["rag pipeline evaluation"],"searchPosts":true,"searchComments":false,"searchCommunities":false,"searchUsers":false,"sort":"relevance","maxItems":7}'

This is a POST endpoint, so the input goes in -i. Here sort is lowercase. The two endpoints spell the same option differently, which is the first thing to copy into your notes. Set maxItems to what you will actually keep: the run billed 7 units for 7 rows, $0.002 each (Verified 2026-10-16).

Merge on the post id

Both endpoints give the post id with the same t3_ prefix, so the merge needs no id conversion:

seen = {}
for post in (c["post"] for c in tikhub["children"]):
    seen[post["id"]] = {"title": post["postTitle"], "url": post["url"], "src": "tikhub"}
for row in apify_rows:
    seen.setdefault(row["id"], {"title": row["title"], "url": row["url"], "src": "apify"})
print(len(seen))  # 14 for our two runs

Our two runs gave 14 unique posts, because no id repeated. If you stored only the TikHub list you would have seen half of them. Before you trust the count, read the duplicate trap in the section on comparing scrapers below.

Can you scrape Reddit yourself with Python?

Yes, and for one subreddit and a few pages it takes an afternoon. You request Reddit's public pages or its JSON, parse the posts out and store them. The same shape of script runs on a schedule if you add a database.

The trouble starts with the thing this article is about. A scraper you write reads one view of Reddit, and the view changes: the ranking moves, the markup changes, and a long run meets rate limits and blocks. To get a list you can trust you end up writing a second reader, a merge on the post id, a duplicate filter, a schedule and a store for every pull, and then you keep all of it working when Reddit changes something. That upkeep is the real cost, not the first script.

Monid is the shorter path for anything you will run more than once. The readers are maintained for you, the output is JSON with the same t3_ ids, and a keyword search is $0.001 a call (Verified 2026-10-16). For a single page that you read by hand, a browser is quicker than any of this.

What does a Reddit search result look like, and what can you build with it?

A Reddit search result is a list of posts, each with a title, a body, a score, a comment count, a subreddit and a link. The fields below are what came back on 2026-10-16.

The TikHub post

With need_format set to true, each entry in children holds a post object with the id (t3_1x78yna), postTitle, url and permalink, content.markdown for the body, score, upvoteRatio, commentCount, createdAt, flair, domain, isSelfPost, authorInfo and a subreddit block with the community name and subscribersCount. Next to children, pageInfo carries endCursor and hasNextPage (true on all three of our calls). Pass the cursor back as after to fetch the next page. No post had a viewCount.

The Apify row

Each row is flat: id, parsedId, url, username, title, communityName, parsedCommunityName, body, html, numberOfComments, upVotes, createdAt, scrapedAt and dataType, plus the booleans isVideo, isAd and over18. There is no upvote ratio. The body and html fields repeat the same text, so drop html before you store a large pull.

What you can build with it

• Keyword monitoring. Run a handful of brand or product terms each day on the per-call endpoint, store the ids, and report only the ids you have not seen. Ten terms a day is about $0.01 a day at $0.001 a call (Verified 2026-10-16). The same monitor looks at people asking for tools in the Reddit user information route when you need the author side.
• A two-source sample. Pull the same term from both endpoints and keep the union. Our union had twice the posts of either list alone.
• A weekly digest. Add time_range: week and sort by TOP to get the best posts of the week for each term, one call per term.
• A comment follow-up. Pick the three posts with the highest commentCount and read only those threads.

What goes wrong when you compare two Reddit scrapers?

Four things showed up in the three result sets of 2026-10-16.

Why does the same post appear twice under two ids?

Because a cross-posted or re-posted thread has a second id. The quoted run returned t3_1wilqoy and t3_1wibufg, both titled "I built rag-eval: A 0-dependency, framework-agnostic RAG pipeline evaluator", posted a few hours apart with different ids. A merge on the id keeps both. If a count matters, also compare normalised titles and authors and keep the older post.

Why does a relevance-sorted list contain an off-topic post?

Because relevance is Reddit's score, not yours. The quoted run returned "Roast My Resume", an image post in r/cscareeradvice, with no mention of RAG in its title or its text. The Apify run returned an interview write-up in r/TechInterviewsIndia whose body mentions production RAG pipelines and evaluation methods and which carried 1,355 upvotes. One is noise and the other is a stretch. Filter on the terms you care about in your own code instead of trusting the order.

Why do two lists of seven look different in age?

Because the search is not limited to the recent past unless you ask. The unquoted TikHub list spanned 2026-04-15 to 2026-10-16. The quoted run ran back to 2024-04-10. Add time_range to make two pulls cover the same window, and always store createdAt so you can see it.

Why do field names break a merge?

Because every source names its fields in its own way. The title is postTitle on one and title on the other, the score is score against upVotes, the comment count is commentCount against numberOfComments, and the body is content.markdown against body. A mapping written by hand once, in one place, is cheaper than a bug in a dashboard.

📖 See also Reddit Scraper: How to Get Reddit Data Without a Key

Which endpoint should I use for which job?

Prices Verified 2026-10-16 with monid inspect. Use the first for a daily sweep of many terms, since a call bills the same however many posts return. Use the second when you want rows in the same shape across a subreddit feed, a post URL and a search, and set maxItems to what you need, because it bills per row.

For other social sources on the same key and balance, browse the Reddit tools.

When is another route a better fit?

You need every post, not a sample. A keyword search returns a ranked page of posts, not a census. If you need all posts in one subreddit, read its feed page by page by URL instead of searching, and store each pull.

You need history. The search covers what Reddit surfaces now. A research question over several years belongs in an archive or dataset built for it, and the rules for using such data differ, so check yours.

You need the conclusion, not the rows. Pulling posts is the cheap half. Reading a few thousand of them to find the complaint that keeps coming back is a summarising job, and tools built for Reddit pain point research do that pass for you.

You need a person, not a post. A username is not an identity. If the job is to find people, not to follow a conversation, a professional network is the right source.

And the disclosure: we resell both endpoints in this article. The measurement is repeatable in four calls, so you can check it yourself.

Conclusion

A Reddit scraper returns different posts because you changed an input or Reddit changed the list, not because it is random. Two identical calls two seconds apart gave the same 7 posts. Two providers gave 7 posts each with none in common. A quoted phrase gave 7 more, with one overlap. The fix is to treat each pull as a dated sample: fix the query, sort and time window, store every pull, and merge sources on the post id.

Two habits matter more than the endpoint you pick. Store createdAt and the pull date with every row, so a change in the list can be told from a change in the source. And dedupe on more than the id, because the same thread can be posted twice.

How we estimated the time saved: writing your own two-source Reddit search with a merge, a duplicate filter and a store is roughly 2 to 4 hours, plus 1 to 4 hours for the first month of upkeep as markup and limits change, and a further hour or two to map the field names. That is 3 to 8 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid discover -q "reddit" to see what is available, monid inspect on the TikHub search to read the schema and the price, and then one call on your own keyword. Start at monid.ai.

FAQ

Published September 2026.