Spark can't find tables in Hive Announcing the arrival of Valued Associate #679: Cesar...
Co-worker works way more than he should
Will I lose my paid in full property
Why does Java have support for time zone offsets with seconds precision?
Did war bonds have better investment alternatives during WWII?
How did Elite on the NES work?
Is there a possibility to generate a list dynamically in Latex?
What's parked in Mil Moscow helicopter plant?
Eigenvalues of the Laplacian of the directed De Bruijn graph
When I export an AI 300x60 art board it saves with bigger dimensions
Is Bran literally the world's memory?
Does a Draconic Bloodline sorcerer's doubled proficiency bonus for Charisma checks against dragons apply to all dragon types or only the chosen one?
Test if all elements of a Foldable are the same
How to compute a Jacobian using polar coordinates?
What was Apollo 13's "Little Jolt" after MECO?
Why did Europeans not widely domesticate foxes?
Determinant of a matrix with 2 equal rows
Is there a way to fake a method response using Mock or Stubs?
Was there ever a LEGO store in Miami International Airport?
How to begin with a paragraph in latex
Is there a verb for listening stealthily?
What is the numbering system used for the DSN dishes?
Coin Game with infinite paradox
When does Bran Stark remember Jamie pushing him?
SQL Server placement of master database files vs resource database files
Spark can't find tables in Hive
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30 pm US/Eastern)Sqoop JDBC connection timeout after import but before hive importWhy is Hadoop not a Data Warehouse?YARN 3 and Spark: allocate a GPUApache Spark 2.4 is not working with yarn cluster manager?Install hadoop before installing SparkIn general,which is faster, Spark or RDMS?Oracle SQL Developer Error when connecting to hive serverMaximind GeoIP2 use in Java + Spark via HDFSimport data from mysql to hive using sqoopThe command spark-shell is throwing a error
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I'm running Hadoop 2.7.7, Hive 2.3.3, and spark-2.4.1-bin-hadoop2.7 all with YARN on a local development system.
I have data in Hadoop that I can read/write with Beeline against Hive no problem. Spark itself is working fine for anything that doesn't try and use Hive.
I can easily reproduce the issue by opening Spark shell and trying to query a table. The table is never found. For testing, everything is in the default database and it does see that, it just doesn't see any tables inside of it.
I have the exact same hive-site.xml in both Spark and Hive so that is not the issue. Nothing errors, it just plain never finds any custom tables and I'm not sure how to even debug it as none of the logs I'm looking at indicate anything is wrong. Any suggestions would be greatly appreciated!
hadoop
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I'm running Hadoop 2.7.7, Hive 2.3.3, and spark-2.4.1-bin-hadoop2.7 all with YARN on a local development system.
I have data in Hadoop that I can read/write with Beeline against Hive no problem. Spark itself is working fine for anything that doesn't try and use Hive.
I can easily reproduce the issue by opening Spark shell and trying to query a table. The table is never found. For testing, everything is in the default database and it does see that, it just doesn't see any tables inside of it.
I have the exact same hive-site.xml in both Spark and Hive so that is not the issue. Nothing errors, it just plain never finds any custom tables and I'm not sure how to even debug it as none of the logs I'm looking at indicate anything is wrong. Any suggestions would be greatly appreciated!
hadoop
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I'm running Hadoop 2.7.7, Hive 2.3.3, and spark-2.4.1-bin-hadoop2.7 all with YARN on a local development system.
I have data in Hadoop that I can read/write with Beeline against Hive no problem. Spark itself is working fine for anything that doesn't try and use Hive.
I can easily reproduce the issue by opening Spark shell and trying to query a table. The table is never found. For testing, everything is in the default database and it does see that, it just doesn't see any tables inside of it.
I have the exact same hive-site.xml in both Spark and Hive so that is not the issue. Nothing errors, it just plain never finds any custom tables and I'm not sure how to even debug it as none of the logs I'm looking at indicate anything is wrong. Any suggestions would be greatly appreciated!
hadoop
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I'm running Hadoop 2.7.7, Hive 2.3.3, and spark-2.4.1-bin-hadoop2.7 all with YARN on a local development system.
I have data in Hadoop that I can read/write with Beeline against Hive no problem. Spark itself is working fine for anything that doesn't try and use Hive.
I can easily reproduce the issue by opening Spark shell and trying to query a table. The table is never found. For testing, everything is in the default database and it does see that, it just doesn't see any tables inside of it.
I have the exact same hive-site.xml in both Spark and Hive so that is not the issue. Nothing errors, it just plain never finds any custom tables and I'm not sure how to even debug it as none of the logs I'm looking at indicate anything is wrong. Any suggestions would be greatly appreciated!
hadoop
hadoop
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
asked 9 hours ago
Michael GrundvigMichael Grundvig
1
1
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Michael Grundvig is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Michael Grundvig is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1428660%2fspark-cant-find-tables-in-hive%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Michael Grundvig is a new contributor. Be nice, and check out our Code of Conduct.
Michael Grundvig is a new contributor. Be nice, and check out our Code of Conduct.
Michael Grundvig is a new contributor. Be nice, and check out our Code of Conduct.
Michael Grundvig is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1428660%2fspark-cant-find-tables-in-hive%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
