Data Rescue Project: “This week’s guest post is from Benjamin Charles Germain Lee, Assistant Professor at the University of Washington, and Kyle Deeds, Assistant Professor at Boston University. Learn more about their recent collaboration to create GovScape, a fantastic resource for searching publicly-available government documents….

We are excited to share GovScape: https://govscape.net, a public search system for 10+ million government PDFs. GovScape is built upon the End of Term Web Archive (https://eotarchive.org/), an incredible multi-institutional effort to document the federal government’s online presence at the end of each presidential administration going back to 2004. GovScape currently includes all renderable PDFs from the 2020 crawl that are 50 pages or under in length. GovScape supports 3 forms of searching over these PDFs:

• keyword search, or exact text search: this form of search is canonical keyword search over document text, i.e., basic keyword search.
• semantic text search, or vectorized natural language text search: with this form of search, you can define more flexible textual queries such as “budgetary data related to the Iraq war” or “rural healthcare for children,” which will return PDF pages ranked based on the relevance of page text to your query – even if the exact string is not present. For this search functionality, we leverage embeddings from the BAAI/bge-base-en-v1.5 model.
• visual search over individual PDF pages: here, you can try queries like “redacted documents,” “pie charts,” or “aerial photography” – PDF pages are returned based on the relevance of their visual features to the query. For this search functionality, we leverage CLIP embeddings generated using the openai/clip-vit-base-patch32 model…”