Here are a few ideas I got:

1. Look up the ship type:

As long as this ship you are seeing is not made by a to date unknown civilication, your databases should be able to identify the ship type, and show you some basic information for that.

2. Estimate by size:

As long as you can measure the distance, you can measure the size of such a ship. Now, your engineers will have an estimate for how much space is taken up by machinery, how much mass is used for the hull, how much space is needed for crew and so on. They know what materials are used commonly, and by multiplying the estimated volume with an average density you can estimate the mass of the ship. It is not a precise estimate, but your engineers should be pretty near to the exact value.

3. Measure the acceleration and energy output of the engines:

Every engine known to us emits radiation in some form due to conversion losses. As long as you can measure the acceleration of such a ship, the radiation of the engines and can identify the type of engine, you should be able to get a good estimate how much energy is used to power the engine, how much force the engines generate and in turn how much mass the ship has.

Everything you need for the above methods are good optical sensors and radiation sensors.

The vacuum of space act as a dielectric.

If you are close enough, provide a net electric charge to your own ship. This will induce a charge also on the probed ship, and will trigger an electrostatic attraction between the two.

Measure your velocity with respect to background and your velocity with respect to the probed ship.

Since you know the charge involved and you can determine the resulting force exerted on the probed ship. The resulting force will change your velocity and the one of the probed ship. You know your mass and your velocity both with respect to background and to probed ship, the only unknown parameter is the probed ship mass.

Probe indirectly

Launch a small probe of precisely known mass so that it passes at a known distance from the ship. The probe might be as opaque and undetectable as could be. At a predetermined moment the probe shoots a low-power laser signal to the mothership, allowing its position to be precisely determined. From there, you can work back the deflection on the trajectory caused by the gravitational mass of the unknown ship.

Probe (almost) directly

Send two probes at different distances from the unknown ship, depending on how sensitive their gravitational sensors are. Measure acceleration on both probes. These will be caused by all nearby masses (the mothership included), but using two probes, the residual that has quadratic dependency on the unknown ship's distance once the other masses are ruled out will give the unknown ship's mass.

For example if all other masses are sufficiently far away, the distance between them and probes P and Q may be taken as constant, so the differential will be zero. The acceleration on probe P will then be given by G(M/a^2 - X/(b+d)^2) while that on probe Q by G(M/(a+d)^2 - X/b^2), with M and X the masses of the mothership and the unknown's, a and b the distances of the first probe from the mothership M and the unknown, and d the distance between the probes:

     M            P         Q                                        X

     |--- a ------|--- d ---|--------------------- b ----------------|

Assuming b, especially, is known with sufficient accuracy (phased laser ranging, maybe?) and accelerometers of sufficient precision are available, the two probes could even be mounted on a fixed "antenna" (it is not necessary for M, P, Q and X to be aligned on a line, but this simplifies things).

Shoot them with a powerful laser and see how much the momentum of their ship changes.

You didn't mention anything about leaving the ship intact.

Shine a laser at it, and measure the (exceedingly slight) change in momentum.

Several years later, in 1922, physicist Arthur Compton performed an
experiment which led to the discovery of the Compton Effect. Proving
Einstein correct, Compton showed that photons indeed have momentum
which is transferrable to materials that have a mass. Compton was
awarded the 1927 Noble Prize in Physics for demonstrating that photons
can transfer their momentum to the electrons with which they collide
inside an atom.

I am, of course, assuming that by the time there are enough spaceships out there that one needs to worry about determining their mass, we will have the technology to be able to detect very minute changes in momentum that a laser beam would impart on a ship. Yes, I am aware that it would be easier to detect the change in momentum caused by a flea smacking into a super jumbo jet, but hey, we are talking about the future here.

Scan them. Fire a very large burst of neutrinos at them and measure how much gets reflected back. In general the more massive an object is the more return signal you will get. It works okay for an estimated order of magnitude.

Unlike other answers this probably won't be recognized as an act of violence, doesn't disturb their trajectory and gives you an answer near to light speeds.

Edit: Actually, I'm no physicist but I think you won't get neutrinos back, you might get electrons, but I'm just not knowledgeable enough to say for sure.

In Civilized Areas, They will Tell You Automatically

Today, ships and aircraft have systems that self report their name, size, exact location, velocity, destination etc. For ships, this is AIS, and for aircraft it is ADS-B. The systems automatically squawk their info a regular intervals, so as soon as you are in receiving range you know what the contact picture is.

It is often required by the relevant authorities, as it significantly improves safety when everyone knows where everyone else is.

Otherwise, IR will give a Good Guess

All vessels will have a power source. This power source is going to have waste heat, which will radiate from the vessel evenly. (Unless the owner pumps air/fluids around to deliberately alter the heat transfer.) This waste heat will be subject to one over r squared losses. If you can determine a range, you can determine how much this blackbody radiation will have dropped off, and you can get an estimate of the size of the power source.

Power is going to be strongly related to vessel size and acceleration. The exact equation is probably going to be complex, but luckily the universe is full of vessels providing you with empirical data!

Gravitational Field, nearly exactly how we are measuring EM field currently. LIGO proved it can be done for large object in 2017 when 2 black holes collided and generated gravitational waves.

Starship in the future should be in the scale of million - trillion tonnes and moving at at least 10-20% speed of light, this would make considerable 'huge' amount of gravitational waves if moving 'close' together (by close I mean a few thousands kilometers). Heck we can measure gravitational waves emitted by black holes light years away now, absolutely we can do that for smaller objects within a few thousands kilometers in 200-300 years, especially if that object moves close to speed of light since it generates more identifiable ripples.

I'm pretty sure we can work the math out to the exact dimensions of LIGO-like version needed to do this, but it would be terrifyingly difficult for people like us. I have strong background in electric engineering and still cannot grasp most part of it.

A patient passive approach could work. Observe ship apparent size and structure (visual angle) relative to your ship. Use an optical distance range finder. Integrate the number of naturally present cosmic rays passing through the other ship into your ship in order to passively estimate density - The ice cube observatory uses similar methods and works for space ships up to almost the thickness of Earth. Get an estimate of the other ships surface area & unseen sides by capacitance (just measure your own ships capacitance, deviation from typical solo deep space value indicates parallel capacitance of the nearby ship). Calculate surface to volume ratio and add density estimate from cosmic ray result for an estimate of mass that improves with more cosmic ray integrations.

Less than a hundreds years ago, we didn't have any application for electromagnetic waves, nor didn't know how to sense it. Now we can detect gravitational waves (well, only the strong). In future, we may have miniaturized sensitive technology to measure the disturbance in space-time.
Who knows, may be we can sense the mass of the ship by measuring the turbulence it causes in the space-time.

You could ask them.

Sure, this wont work in every situation. But if they cooperate, it's the best way. Your communications range is probably further than your sensor range, and you'll get the most accurate possible answer. Their computer will have to have the exact mass measurement at all times.

Equipment that can survey the gravitational potential on the order of microgals is quite common today in geological industry and academia, known as gravimetry.

The machines used as instrumentation, gravimeters, are quite interesting, but ultimately they are able to derive densities by way of directly measuring differing gravitational acceleration at many points. These have even been used to probe non-geological manmade-scale objects e.g. when French scientists surveyed the Cheops pyramid for unknown chambers.

A tech advanced gravimeter (perhaps using an array of atom chip nano gravimeters) paired with a good gravimetric map of the navigation area should allow measurement of the mass of an anomalous body. With enough resolution you could even possibly map the density distribution of the vessel and track masses moving on the interior.