Tjyylli

Reading the first amendment strictly through a textualist lens it is understood that Congress cannot limit our speech, but the first amendment does not say anything about companies limiting it, or even the Executive branch limiting our freedom of expression through an executive order.

First Amendment:

Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances.

Since the first amendment text is only about Congress's restrictions on making laws about speech, is the understanding that "we have the freedom of speech" based off of precedent set by courts?

Trivially, yes

The first amendment was adopted on December 15, 1791.

Every time there has been a dispute about what it means that has gone to court since then, the judgement of that court has established, overturned or clarified precedent - that's what common law courts do.

The government can limit your speech

The Supreme Court has recognized categories of speech which receive lesser or no protection from the first amendment. For example, inciting lawless actions, fighting words, true threats, obscenity, child pornography etc.

They have also determined that it doesn't limit the government's power to impose reasonable time, place or manner restrictions on speech. As Justice Holmes put it in Schenck v. United States (1918), "Even the most stringent protection of free speech would not protect a man in falsely shouting fire in a theater and causing panic."

It applies to parts of government which derive their power from Congress

Which is, in most cases, all government.

The executive actually has surprisingly little power granted by the Constitution (Article II, Section 2). All the other powers of the executive are technically delegated powers of Congress and are therefore subject to the first amendment.

Similarly, only the Supreme Court draws its mandate without going through Congress Article III, Section 1) - all other courts are subject to first amendment restrictions.

It only restricts government

The limitation is a negative one on the US Congress (and through incorporation, the states). It does not, of itself, restrict private actors who are free to restrict speech however they want within their own property, including both physical and online spaces.

It is open to the government to enact laws that would extend an affirmative right to free speech onto non-state actors (see Pruneyard Shopping Center v Robins (1980)), however, the Federal government has not done so and neither have most states.