A few months ago I promised we would be launching a new version of SymbolSource. Today I’m very happy to make the first part of the new service public, the one you’ve been waiting for most impatiently.
A new package repository has been deployed to production at https://nuget.smbsrc.net and contains all packages from https://www.symbolsource.org/Public/Metadata/NuGet (around 50 thousand of them!). In other words this is the new complementary service to https://www.nuget.org, running on a codebase rewritten from scratch – something we needed to do to solve all of the performance issues you have been facing with SymbolSource in the past.
What about the SymbolSource repository for MyGet, or the other company instances, you might ask? Those will be migrated to the new service too, and I will report as soon as we will have done that.
For now let’s see how the new service operates. You can try all of these commands yourself, and update all of your scripts and debuggers accordingly.
Publishing a symbol package
To push a symbol package run this command:
> nuget push SymbolSource.DemoLibrary-1.0.0.0.nupkg a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net Pushing SymbolSource.DemoLibrary 1.0.0.0 to 'https://nuget.smbsrc.net'... Your package was pushed.
The command will succeed immediately without any timeouts, it just sends data to Azure storage. Packages are queued and processed asynchronously. That means that you will need to wait a bit before the symbols and sources become available, but we’ll be monitoring the queue length to make sure it never takes more than a few minutes.
Waiting and refreshing isn’t much fun, so we included a simple notification service, which you can use directly from the command prompt. Just append your Twitter username to the API key after a slash, like this:
> nuget push SymbolSource.DemoLibrary-1.0.0.0.nupkg a0f4e24d-851f-4327-a9ec-181d335e7e95/@TripleEmcoder -source https://nuget.smbsrc.net Pushing SymbolSource.DemoLibrary 1.0.0.0 to 'https://nuget.smbsrc.net'... Your package was pushed.
You’ll get tweeted by our friendly SymbolSource Bot (@smbsrc) when the package starts processing and when it finishes. The tweet will also tell you if anything went wrong, in which case @SymbolSource will also be mentioned to alert us about the issue.
One thing you’ll notice immediately is that we’ve added a suffix to the version number. Each upload to SymbolSource is identified separately. We then do a form of reference counting on all of the symbols and sources, so there is no cost to that. But in case you upload different builds with the same version number, all of them will be available to clients. Unless of course you decide to delete a package. Which is now a fully supported self-service operation!
Deleting a symbol package
If you know the SymbolSource version string of your package, deleting is as simple as running this command:
> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95/@TripleEmcoder -source https://nuget.smbsrc.net Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net'. SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 was deleted successfully.
Once again NuGet will lie a bit, as this is an asynchronous operation. But you will get the same friendly tweets if you include your username with the API key, as I did above.
But what if you didn’t push with Twitter notifications enabled, and are now at a loss as to what the version string is? No worries, you can query SymbolSource as any other package source and find out what the version is.
Before we talk about that, however, if you really want to remove your content from SymbolSource, run this too:
> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net/,all,original Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net'. SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 was deleted successfully.
SymbolSource saves a copy of all uploaded packages, in case there is a change in the indexing algorithm and we need to reprocess. They are hosted in subfeed called original – see below for an explanation of what that is.
Listing symbol packages
By now you probably have already guessed the command to list packages from SymbolSource:
> nuget list SymbolSource.DemoLibrary -source https://nuget.smbsrc.net No packages found.
Well, that result might seem unexpected. But remember that we added an additional version string and therefore promoted all packages to semantic versioning. NuGet treats those as prerelease packages. A quick fix:
> nuget list SymbolSource.DemoLibrary -prerelease -source https://nuget.smbsrc.net SymbolSource.DemoLibrary 1.0.0.0-at-0E8GM1
There it is! Just remember about that NuGet will only show the most recent package, but you can show all of the uploads:
> nuget list SymbolSource.DemoLibrary -prerelease -allversions -source https://nuget.smbsrc.net SymbolSource.DemoLibrary 1.0.0.0-at-0E2NAN SymbolSource.DemoLibrary 1.0.0.0-at-0E8G1E SymbolSource.DemoLibrary 1.0.0.0-at-0E8GM1
All of this works in NuGet Package Explorer too! As an exercise try downloading one of those packages. What you will get is a status package, useful in determining what symbols and source actually have been indexed by SymbolSource. More on this next time.
And in case you’re wondering… That version string is just a timestamp, encoded into an alphabet-based number system to save length. Unfortunately NuGet imposes a 20-character limit on all semantic versions.
Advanced listing by status and ownership
In my last post I also mentioned subfeeds, which let you query by status and ownership. To recap here are all of the statuses – they’ve slightly changed since our original spec:
new– not yet opened, random package name and unknown version,original– a copy of the uploaded package, without any processing, useful if we improve the indexer and need to resubmit packages without any users’ actions,indexingqueued– indexing will soon start asynchronously,indexing– processing is in progress as you are looking at the list,succeeded– the package has been indexed without any issues,deletingqueued– deleting with soon start asynchronously,deleting– deleting is in progress as you are looking at the list,deleted– the package has been completely removed from the index,partial– something went wrong, either during indexing or deleting, and some symbols and sources might be available, but some may not,damagednew– package could not be read with a standard NuGet library,damagedindexing,damageddeleting– these will probably indicate an error in SymbolSource.
For example, if you wanted to confirm that a package has been deleted, you would need to run this command:
> nuget list SymbolSource.DemoLibrary -prerelease -allversions -source https://nuget.smbsrc.net/,all,deleted SymbolSource.DemoLibrary 1.0.0.0-at-0D9I8Z SymbolSource.DemoLibrary 1.0.0.0-at-0D9I90 SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51
As you can see, the list contains a package we did delete a few paragraph earlier – version 1.0.0.0-at-0E8H51. There are however also some old versions, which I deleted before writing this post. I can now delete them again, sort of, to simply remove the status package. Note that at this point symbols and sources have already been deleted, this is just similar to cleaning log files:
> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net/ Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net/'. Failed to process request. 'Not Found'. The remote server returned an error: (404) Not Found..
A 404 error is a bit odd, isn’t it? Well not exactly. Remember, we have already deleted this package. It doesn’t show in the feed. To remove the status package we need to address the subfeed specifically.
> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net/,all,deleted Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net/,all,deleted'. SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 was deleted successfully.
If you don’t specify a subfeed at all, SymbolSource defaults to succeeded state and all packages (as opposed to own packages). The first comma separates the feed name, which in case of nuget.org is empty. Things will be different for myget.org, but more on that in a future post.
So far we have listed all packages, regardless of who uploaded them. Of course pushing and deleting is permitted according to nuget.org package ownership. But you can also limit the packages listed by a query, using the following command:
nuget list -prerelease -allversions -source https://nuget.smbsrc.net/,own,succeded Please provide credentials for: https://nuget.smbsrc.net/,own,succeded UserName: @TripleEmcoder Password: ************************************ SymbolSource.DemoLibrary 1.0.0.0-at-0E2NAN SymbolSource.DemoLibrary 1.0.0.0-at-0E8G1E SymbolSource.DemoLibrary 1.0.0.0-at-0E8GM1
SymbolSource will ask for credentials this time, as it needs to known who the user is, to list only the right packages. You can pass anything as the username, but if you give us something meaningfull like a Twitter handle, we will be able to get back to you in case of problems. The password is your NuGet API key.
Debugging in Visual Studio
And last, but not least, the new URL for Visual Studio is…
https://nuget.smbsrc.net
Just replace the old symbolsource.org URL in your Visual Studio options and you’re done. Happy (and fast) debugging 🙂
Wrapping it up
In this post I described the core features of our new SymbolSource engine. I hope you will find the new service much faster and much more stable than the previous one. We will be gradually upgrading other parts of the service over the next weeks and months. If you have any questions or feedback please leave them in the comment section.
Marcin Mikołajczak a.k.a. TripleEmcoder 