Using SVGs as images is not possible for a number of reasons:
SVGs don’t have a defined “size” like bitmap images. They may define a viewbox so it is clear what aspect ratio is needed to display it properly and what the maximum x/y coordinates will be – but the real size is not fixed, it can be as big or as small as needed. For example:
https://dev.w3.org/SVG/tools/svgweb/samples/svg-files/android.svg
https://dev.w3.org/SVG/tools/svgweb/samples/svg-files/basura.svg
Some SVGs don’t have a viewport and a human has to decide how big the visible part must be, so that all content is displayed:
https://dev.w3.org/SVG/tools/svgweb/samples/svg-files/tiger.svg
https://dev.w3.org/SVG/tools/svgweb/samples/svg-files/DroidSansMono.svg
So it is just impossible to define at which size an SVG should be displayed when opening it in PhotoSwipe.
And finally SVGs are not only “images” – SVGs can also contain JavaScript to create interactive elements:
https://rawgit.com/petercollingridge/code-for-blog/master/svg-interaction/interactive-map/colour_with_class.svg
http://atomicrobotdesign.com/blog_media/svg_mask.html
The scripting part of SVGs is the main reason why WordPress does not allow to upload SVGs anyway.
A proper way to embed SVG would be to embed SVG into a DIV element and use this DIV element as part of a gallery since PhotoSwipe itself allows to embed HTML instead of images (see for example https://photoswipe.com/documentation/custom-html-in-slides.html). However even this would require a lot of extra work to be done and even then it may be that the SVG does not have a viewport and can not just be zoomed to full screen size.
Eventhough I don’t plan to add this anytime soon, pull requests are welcome if anyone wants to add this feature:
https://github.com/arnowelzel/lightbox-photoswipe