Recovering Nexus Repository After H2 Database Corruption#

Initial Checks#

From a client machine, the root page and static assets responded:

1
curl -I https://nexus.mcfpp.top/
2
curl -I https://nexus.mcfpp.top/static/rapture/bootstrap.js

But Nexus REST and repository endpoints timed out:

1
curl -m 20 -v https://nexus.mcfpp.top/service/rest/v1/status/check
2
curl -m 20 -v https://nexus.mcfpp.top/service/rest/v1/repositories
3
curl -m 20 -v https://nexus.mcfpp.top/repository/maven-releases/

On the server, the same thing happened locally:

1
curl -m 10 -v http://127.0.0.1:8081/service/rest/v1/status/check

It connected to port 8081, sent the HTTP request, and then received no response until timeout. That ruled out Gradle and mostly ruled out nginx. Nexus was hung internally.

Restart Exposed the Real Failure#

Restarting Nexus did not cleanly stop the old JVM:

1
nexus.service stop-sigterm timed out. Killing.
2
main process exited, code=killed, status=9/KILL

After restart, Nexus exited with status 255, and nginx started returning 502 Bad Gateway. That was expected: nginx was alive, but the Nexus backend was dead.

journalctl only showed the wrapper process. The useful error was in Nexus’s own log:

1
sudo tail -n 300 /opt/sonatype-work/nexus3/log/nexus.log

The key line was:

1
org.h2.jdbc.JdbcSQLNonTransientConnectionException:
2
File corrupted while reading record:
3
"/opt/sonatype-work/nexus3/db/nexus.mv.db".
4
Possible solution: use the recovery tool [90030-232]
5

6
Caused by: org.h2.mvstore.MVStoreException:
7
File is corrupted - unable to recover a valid set of chunks [2.3.232/6]

So the actual root cause was:

1
/opt/sonatype-work/nexus3/db/nexus.mv.db was corrupted.

Before Touching Anything: Take a Full Backup#

Before repair attempts, stop Nexus and copy the whole data directory:

1
sudo systemctl stop nexus
2

3
sudo tar -C /opt/sonatype-work \
4
  -czf /root/nexus3-before-h2-recovery-$(date +%F-%H%M%S).tar.gz \
5
  nexus3

This matters because H2 recovery can make things worse if run against the only copy.

H2 Recovery Attempt#

The Nexus log mentioned H2 version 2.3.232, so I downloaded the matching H2 jar:

1
cd /root
2
curl -fL -o h2-2.3.232.jar \
3
  https://repo.maven.apache.org/maven2/com/h2database/h2/2.3.232/h2-2.3.232.jar

Because the nexus user cannot normally read files under /root, the jar had to be copied somewhere readable:

1
sudo cp /root/h2-2.3.232.jar /opt/sonatype-work/nexus3/db/
2
sudo chown nexus:nexus /opt/sonatype-work/nexus3/db/h2-2.3.232.jar
3
sudo chmod 644 /opt/sonatype-work/nexus3/db/h2-2.3.232.jar

Then I tried H2’s recovery tool:

1
cd /opt/sonatype-work/nexus3/db
2

3
sudo -u nexus /opt/nexus/jdk/temurin_17.0.13_11_linux_x86_64/jdk-17.0.13+11/bin/java \
4
  -cp /opt/sonatype-work/nexus3/db/h2-2.3.232.jar \
5
  org.h2.tools.Recover -dir . -db nexus

In this incident, recovery failed:

1
org.h2.mvstore.MVStoreException:
2
File is corrupted - unable to recover a valid set of chunks [2.3.232/6]

At that point, the corrupted H2 file was not directly recoverable.

Getting Nexus Back Online#

The practical recovery path was to get Nexus started again with a clean database, while keeping the old blob store content.

The important directories are:

1
/opt/sonatype-work/nexus3/db
2
/opt/sonatype-work/nexus3/blobs
3
/opt/sonatype-work/nexus3/etc

The database contains Nexus metadata: repositories, users, components, assets, tasks, and configuration. The blob store contains the actual artifact files.

After Nexus started again, the UI initially still looked stuck because the browser had cached stale redirects/assets. A hard refresh fixed that:

1
Ctrl+F5

Opening the site in a private/incognito window is also a good check.

Recreating Repository Metadata From Blob Store#

After the UI was available again, the Maven repositories existed but looked empty. Running these tasks first did not bring old components back:

1
Repair - Rebuild Maven repository metadata
2
Repair - Rebuild repository browse
3
Repair - Rebuild repository search

Those tasks rebuild indexes and Maven metadata from the current database. They do not recreate missing component rows if the database was lost or recreated.

The task that actually mattered was:

1
Repair - Reconcile component database from blob store

In the Nexus UI:

1. Open Administration.
2. Go to System -> Tasks.
3. Create task.
4. Choose Repair - Reconcile component database from blob store.
5. Select the blob store that contains the old artifacts, likely default.
6. Run it for the hosted Maven repositories, especially maven-releases and maven-snapshots.

After that task completed, the old components appeared again and downloads worked.

Then I reran:

1
Repair - Rebuild Maven repository metadata
2
Repair - Rebuild repository browse
3
Repair - Rebuild repository search

This rebuilt the Maven metadata, browse tree, and search index after the component database was repopulated.

Sonatype documents this recovery task as the one used when blob storage and the component database are out of sync:

1
https://help.sonatype.com/en/tasks.html

Confirming Old Blob Data Exists#

These commands are useful before or during reconciliation:

1
sudo du -sh /opt/sonatype-work/nexus3/blobs/*
2

3
sudo find /opt/sonatype-work/nexus3/blobs \
4
  -type f -name '*.properties' | wc -l
5

6
sudo grep -RIl '@Bucket.repo-name=maven-releases' \
7
  /opt/sonatype-work/nexus3/blobs/*/content | head
8

9
sudo grep -RIl '@Bucket.repo-name=maven-snapshots' \
10
  /opt/sonatype-work/nexus3/blobs/*/content | head

If those grep commands find files, the artifacts are probably still present in the blob store. If they find nothing, the old blob content may be gone, stored under another blob store, or associated with different repository names.

Reverse Proxy and Browser Notes#

During recovery, I also saw a browser error like this:

1
GET https://nexus.mcfpp.top/static/rapture/app.js?... net::ERR_TOO_MANY_REDIRECTS

From curl, the same asset returned 200 OK, so this was browser state, not an active server redirect loop. A hard refresh fixed it.

Still, for Nexus behind HTTPS nginx, the proxy should preserve the original host and scheme:

1
proxy_set_header Host $host;
2
proxy_set_header X-Real-IP $remote_addr;
3
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
4
proxy_set_header X-Forwarded-Proto https;
5
proxy_set_header X-Forwarded-Host $host;

The Nexus Base URL capability should also point at the public HTTPS URL:

1
https://nexus.mcfpp.top/

Final Verification#

After reconciliation, verify the service locally:

1
curl -m 10 -v http://127.0.0.1:8081/service/rest/v1/status
2
curl -m 10 -v http://127.0.0.1:8081/service/rest/v1/status/check

Verify the public Maven repository:

1
curl -I https://nexus.mcfpp.top/repository/maven-releases/

Then retry publishing:

1
./gradlew publish --stacktrace

Lessons Learned#

First, a Nexus UI stuck on Initializing ... does not necessarily mean a frontend problem. In this case, the static UI loaded, but REST and repository endpoints were hung because the embedded H2 database was corrupted.

Second, journalctl may not contain the real exception. For Nexus, always check:

1
/opt/sonatype-work/nexus3/log/nexus.log
2
/opt/sonatype-work/nexus3/log/jvm.log
3
/opt/sonatype-work/nexus3/log/karaf.log

Third, the important recovery distinction is:

1
Rebuild metadata/search/browse:
2
  Rebuilds derived data from the database.
3

4
Reconcile component database from blob store:
5
  Recreates missing component metadata from blob files.

If the database was recreated or restored empty, the reconcile task is the key.

Finally, backups need to cover both metadata and blobs:

1
/opt/sonatype-work/nexus3/db
2
/opt/sonatype-work/nexus3/blobs
3
/opt/sonatype-work/nexus3/etc

For a self-hosted Maven repository, losing the database is painful, but it is not necessarily fatal if the blob store is still intact and the repository names/blob store mapping can be recreated.