Publishing is not the hard part in this city. Institutions here generate pages by the tens of thousands without anyone deciding to. The hard part is choosing which of those pages a search engine should ever be asked to look at.
Nearly every site with a discovery problem has the same self-image: the content exists, the site is live, the pages are linked from somewhere, so the work is done. Then a page that took a month to produce collects nothing for eleven weeks, and the diagnosis begins in the wrong place — with keywords, with links, with the copy — when the actual failure happened earlier and more quietly.
Between writing a page and receiving a visitor there are three separate events, and they fail independently. A page has to be discovered, then crawled, then judged worth keeping in an index. Teams treat this as one step called "getting indexed." It is not, and the middle stage is where institutional publishing in Greater Boston runs into trouble.
Existing and being found are different states
A crawler reaches a page along one of a few routes: a link from a page it already knows, an entry in a sitemap, a direct submission, or a reference from another site. Absent all four, the page is invisible regardless of quality. This sounds obvious until you look at how much modern content sits behind a filter, a search box or a JavaScript tab, reachable by a human and by nothing else.
Each arrow between those states has its own failure mode, and each fails silently. Nothing notifies you when a template's worth of URLs is discovered and never fetched. Nothing announces that ninety percent of a directory was crawled and dropped. The absence shows up as flat impressions on a section you assumed was working, which is why it can persist for a year.
Known but never fetched
The URL is on record and no bot has spent a request on it, usually because thousands of similar URLs sit ahead of it in the queue.
- Looks identical to a ranking problem
- Fixed by pruning, not by writing
Fetched and discarded
A bot arrived, read the page and declined to store it, which is a verdict on the content rather than on your architecture.
- Common on generated near-duplicates
- Resubmission changes nothing
Crawl budget, and who is spending yours
Crawl budget is the amount of fetching a search engine is willing to do on your host in a given period. It is not a published figure and it is not something you buy. It rises with a site's demonstrated value and its ability to respond quickly, and it falls when a crawler repeatedly spends requests on pages that turn out to be worthless.
Small sites rarely hit the ceiling. Institutions hit it constantly, and in this market institutions are the norm rather than the exception. Consider what a mid-sized academic medical center publishes without a single editorial decision being made.
Clinician and faculty directories
One profile per person, plus a URL for every combination of specialty, location, language and insurance filter.
- Thousands of profiles as a base
- Filters multiply, they do not add
- Sort orders duplicate every result set
Publication and citation lists
Every author, every year, every journal, frequently paginated twenty items at a time and reachable three ways.
- The same paper on four URLs
- Author pages with one item each
- Year archives back two decades
Course and program catalogs
Each course multiplied by each term, retained after the term ends because nobody built a retirement rule.
- Ten years of expired sections
- Near-identical descriptions
- Cross-listings under several codes
Event and seminar archives
Weekly colloquia, grand rounds and lab meetings, each with a permanent page and a calendar view per month.
- Fifty pages a year, per series
- Day, week and month views
- Speaker pages with no content
None of this is anyone's fault. A directory template that produces a URL per filter combination is good engineering for humans and an arithmetic disaster for crawling. Four thousand profiles crossed with three sort orders and five filter states is not four thousand pages. It is sixty thousand, and they are all real URLs returning real content with a status of 200.
Which templates deserve to be crawled at all
The instinct on discovering forty thousand uncrawled URLs is to get them crawled. That is almost always the wrong instinct. The better question is which of the generating templates should be producing indexable pages in the first place, because a template is a decision made once and executed thousands of times.
Evaluate a template, not a page. For each family of generated URLs, ask three things: does anyone search for what this page uniquely contains, does the page answer that search without a human intermediary, and would the person landing on it have any reason to continue. A faculty profile passes all three — people search names, the page answers, and the next step is contact. A calendar month view from 2016 fails all three and there are one hundred and twenty of them.
| Template family | Typical volume | Search demand | Recommended handling |
|---|---|---|---|
| Individual clinician or faculty profiles | Hundreds to thousands | High, name-driven and specific | Index, sitemap, keep current |
| Filter and sort permutations | Tens of thousands | None | Block from crawling, canonicalize to the base list |
| Protocol, trial and method pages | Dozens to hundreds | High, technical and evidence-seeking | Index and prioritize; this is your best material |
| Publication list pagination | Thousands | Near zero beyond page one | Leave crawlable, keep out of sitemaps |
| Expired course sections | Thousands | Minimal after the term closes | Retire on a schedule, keep the parent course |
| Event archive, past dates | Thousands | Low except for named speakers | Keep a curated subset, no-index the rest |
The protocol and method pages in that table deserve a note of their own. In a market where the searcher often holds a doctorate in the subject, the technical pages nobody in marketing wants to touch are frequently the highest-converting documents a site owns. They lose their crawl priority to a directory template that produces sixty thousand URLs of nothing. Fixing that ratio is worth more than any amount of new content.
A sitemap is an instrument, not paperwork
A sitemap is usually treated as a compliance artifact: generate it once, submit it, forget it. Read properly it is the most direct statement you can make about which of your URLs matter, and the submission tooling in the panel treats it that way: as a curated list you hand over deliberately, rather than leaving a crawler to find its own way through a filter interface.
Recursive parsing, three levels deep
Built for sites whose URL inventory arrives as an index of indexes rather than a single file.
- Upload or point at a URL. Submit the file directly or give the address of a live sitemap and let the parser retrieve it.
- Three levels of nesting. An index referencing indexes referencing sitemaps is followed all the way down without manual expansion.
- Up to 1,000 sitemaps in one job. Large institutional inventories can be handed over in a single operation rather than file by file.
- Two jobs at once, twenty waiting. Two sitemap jobs run concurrently with up to twenty queued, so a portfolio can be processed in sequence unattended.
Three levels matters more than it sounds. A university web estate typically has a root index pointing at departmental indexes pointing at the sitemaps a dozen separate content systems produce. Flattening that by hand is a morning's work that has to be repeated whenever a department redesigns. Following it automatically means the structure the institution actually has is the structure you submit.
Direct submission and what the numbers mean in practice
Beyond sitemaps sits direct submission: naming URLs individually and pushing them toward the crawlers. The limits are specific, and reading them as arithmetic rather than as features changes how you plan.
A daily budget and a batch ceiling
A tracker with a fixed daily allowance per account, and bulk operations for large inventories.
- 1,000 URLs per day, per account. The allowance is shared across every property you manage, which makes it a scheduling constraint before it is a technical one.
- 10,000 URLs per batch. A batch can be prepared in one operation; the daily budget governs how quickly it drains.
- Delivered through IndexNow. Submission reaches GoogleBot and BingBot through the IndexNow API rather than through manual per-URL requests.
- Logged individually. Each URL carries its own record: bot visit with timestamp, current status and error detail when something failed.
Now do the arithmetic against the directory from earlier. Sixty thousand generated URLs, at a thousand a day, is sixty days of a shared account budget spent entirely on filter permutations that no person has ever searched for. During those two months every genuinely valuable page you publish waits behind them. That is the cost of not pruning, expressed in a unit anyone can understand.
The same arithmetic run the other way is encouraging. Four hundred protocol and method pages, eleven hundred clinician profiles and two hundred program pages come to roughly seventeen hundred URLs — under two days of budget for the entire set of documents your organization actually wants found. Selection is not an austerity measure here. It is the difference between two days and two months.
What a batch is telling you while it runs
Submission without observation is superstition. The value of a per-URL log is that it converts an act of faith into a measurement, and three live counters carry most of the signal: how many URLs were submitted, how many were found, and how many failed.
Failures are the useful ones, because they are specific. A URL that returns a server error, redirects into a chain, or is blocked by your own directives will say so, with a timestamp. That is a technical defect list, and it is generally short and fixable — far more actionable than a vague sense that indexing is slow.
The gap between submitted and found is the diagnostic that matters most. If a batch of two thousand profile pages shows near-complete bot visits within a week while a batch of two thousand event archive pages barely moves, you have not discovered a bug. You have received an opinion. The crawler is telling you which template it considers worth its time, and it is usually right.
| Pattern in the log | Most likely cause | What to do |
|---|---|---|
| High failure rate, one template | Server errors or a directive blocking the path | Fix the technical fault before resubmitting anything |
| Submitted high, found low, no errors | The pages are reachable but not judged worth fetching | Reconsider whether the template should be indexable |
| Found quickly, impressions still zero | Crawled and either dropped or ranked far down | A content and intent problem, not a discovery problem |
| Sitemap job accepted, few URLs registered | Nesting deeper than three levels, or unreachable child files | Flatten the structure or submit the child indexes directly |
Running this across a portfolio
Institutions rarely operate one site. A hospital system has the main domain, a research institute, a foundation, a handful of department microsites and whatever a lab built in 2019 and never handed over. Agencies serving this market have the same problem in a different shape.
- Group the accounts, not the logins. Linked Google account groups keep several properties under one workspace instead of a spreadsheet of credentials.
- Share single sites by email. A department can be given access to its own property without seeing the rest of the estate.
- Tag sites and filter globally. Site tags act as a filter across the whole panel, so a client group or a division can be isolated in one action.
- Queue the sitemap work. Two jobs run at once with twenty more waiting, which is enough to process an entire portfolio overnight without supervision.
- Remember the budget is shared. One thousand URLs a day belongs to the account, not to each site — allocate it deliberately or the noisiest property will absorb it.
That last point is the one that catches teams managing several domains. A single unpruned template on one property will quietly consume the daily allowance for every other property in the group. The Indexing Hub makes the consumption visible, which is the first step toward deciding it deliberately rather than discovering it in a quarterly review.
Questions that come up
How long should I wait before deciding a submitted page will not be indexed?
Give it several weeks, and check the log before you conclude anything. If the record shows a bot visit and the page still collects no impressions, that is an indexing or ranking judgement, not a discovery failure. If no bot ever arrived, the problem is upstream and resubmitting will not solve it.
We have 90,000 URLs and a budget of 1,000 a day. Where do we start?
Not with submission. Sort the inventory by template and decide which families deserve to be indexed at all. Most estates of that size reduce to two or three thousand genuinely valuable URLs, which is a few days of budget. The remaining eighty-seven thousand need a decision, not a queue slot.
Should we delete the old event and course archive?
Rarely delete; usually stop advertising. Keep the pages reachable for anyone who holds a link, remove them from your sitemaps, and no-index the families with no demonstrated search demand. Deleting breaks citations, and in an academic setting those citations are often the reason the domain has authority in the first place.
Does submitting a page more than once help it along?
No, and it spends budget you need elsewhere. Resubmission is appropriate when a page has genuinely changed or when a technical fault that blocked the fetch has been repaired. Repeated submission of an unchanged page communicates nothing new.
Our sitemap job finished but far fewer URLs appeared than we expected.
Check the nesting depth first, since parsing follows three levels. An estate with four or five layers of index files will lose everything below the third. Submit the deeper index files directly, or flatten the structure so the whole inventory sits within reach.
A calculation worth running before you generate anything else
Take one generated template — the event archive, the filter permutations, the expired course sections — and count the URLs it produces. Then open ninety days of Search Console filtered to that path and count the clicks. Divide. Most institutional sites in this region have at least one template producing thousands of URLs and fewer than fifty clicks a quarter.
That ratio is the whole argument. Those pages are not free: they consume crawl capacity, they dilute the signal of the pages that matter, and they occupy a daily submission budget that a protocol page or a clinician profile could have used. Removing them from the index costs nothing and returns capacity immediately.
The pattern in this market is consistent enough to state plainly. The pages worth finding are the specific ones — the method, the protocol, the named person, the actual program — and they are usually outnumbered fifty to one by pages a system generated because nobody told it to stop. Curating that ratio is unglamorous work with a larger effect than most content projects. Our technical SEO services cover how that audit runs on an institutional estate, and the blog archive takes the analytics side further.
If you want the inventory in front of you rather than in your imagination, submit your sitemap index and watch which templates the crawlers accept and which they ignore: open the Semalt dashboard and connect your site. Automated campaign work sits alongside it — AutoSEO at $149 per month per domain for keyword discovery and link building, or the FullSEO tier at $500 per month per domain with a team of specialists, developers and writers behind it, with first measurable movement typically four to eight weeks out. But run the template calculation first. It usually changes what you would have asked for.
